All of RobinHanson's Comments + Replies

Talk by Robin Hanson on Elites

Aw,  I still don't know which face goes with the TGGP name.

1teageegeepea2moI was wearing a shirt designed by one of your colleagues.
Robin Hanson's Grabby Aliens model explained - part 1

Wow, it seems that EVERYONE here has this counter argument "You say humans look weird according to this calculation, but here are other ways we are weird that you don't explain." But there is NO WAY to explain all ways we are weird, because we are in fact weird in some ways. For each way that we are weird, we should be looking for some other way to see the situation that makes us look less weird. But there is no guarantee of finding that; we just just actually be weird.

2rodarmor2moWhat do you think about my counter-argument here, which I think is close, but not exactly of the form you describe: [] To sum up "You say humans look weird according to this calculation and come up with a model which explains that, however that model makes humans look even more weird, so if the argument for the model being correct is that it reduces weirdness, that argument isn't very strong."
Robin Hanson's Grabby Aliens model explained - part 1

You have the date of the great filter paper wrong; it was 1998, not 1996.

9Writer2moRobin Hanson, you know nothing about Robin Hanson. You first wrote the paper in 1996 and then last updated it in 1998. ... or so says Wikipedia [], that's why I wrote 1996. I just made this clear in the video description anyway, tell me if Wikipedia got this wrong. Btw, views have nicely snowballed from your endorsement on Twitter, so thanks a lot for it.
Grabby aliens and Zoo hypothesis

Yes, a zoo hypothesis is much like a simulation hypothesis, and the data we use cannot exclude it. (Nor can they exclude a simulation hypothesis.) We choose to assume that grabby aliens change their volumes in some clearly visible way, exactly to exclude zoo hypotheses. 

4avturchin9moIf we are inside the colonisation volume, its change will be isotropic and will not look strange for us. For example, if aliens completely eliminated stars of X class as they are the best source of energy, we will not observe it, as there will be no X stars in any direction.
Highlights from the Blackmail Debate (Robin Hanson vs Zvi Mowshowitz)

Your point #1 misses the whole norm violation element. The reason it hurts if others are told about an affair is that others disapprove. That isn't why loud music hurts.

2ESRogs1ySuppose it were easy to split potential blackmail scenarios into whistleblower scenarios (where the value of the information to society is quite positive) and embarrassing-but-useless scenarios (where it is not). Would you support legalizing blackmail in both classes, or just the first class? EDIT: I ask because, I think (at least part of) your argument is that if we legalize paying off whistleblowers, then that's okay, because would-be-whistleblowers still have an incentive to find wrongdoing, and the perpetrators still have an incentive to avoid that wrongdoing (or at least hide it, but hiding has costs, so on the margin it should mean doing less). (This reminds me a bit of white hat hackers claiming bug bounties.) Meanwhile, the anti-blackmail people argue that you don't want people to be incentivized to find ways to harm each other. So, if you could cleanly separate out the public benefit from the harm, on a case-by-case basis (rather than having to go with simple heuristics like "gossip is usually net beneficially"), it seems like you might be able to get to a synthesis of the two views.
Highlights from the Blackmail Debate (Robin Hanson vs Zvi Mowshowitz)

Imagine there's a law against tattoos, and I say "Yes some gang members wear them but so do many others. Maybe just outlaw gang tattoos?" You could then respond that I'm messing with edge cases, so we should just leave the rule alone.

6Vaniver1yA realistic example of this is that many onsen [] ban tattoos as an implicit ban on yakuza, which also ends up hitting foreign tourists with tattoos. It feels to me like there's a plausible deniability point that's important here ("oh, it's not that we have anything against yakuza, we just think tattoos are inappropriate for mysterious reasons") and a simplicity point that's important here (rather than a subjective judgment of whether or not a tattoo is a yakuza tattoo, there's the objective judgment of whether or not a tattoo is present). I can see it going both ways, where sometimes the more complex rule doesn't pay for itself, and sometimes it does, but I think it's important to take into account the costs of rule complexity.
Analyzing Blackmail Being Illegal (Hanson and Mowshowitz related)

You will allow harmful gossip, but not blackmail, because the first might be pursuing your "values", but the second is seeking to harm. Yet the second can have many motives, and is mostly commonly to get money. And you are focused too much on motives, rather than on outcomes.

1curi1yIf I threaten to do X unless you pay me, then the motive for making that threat is getting money. However, I don't get money for doing X. There are separate things involved (threat and action) with different motives.
1Ericf1yOk. I don't think that's the central example of what people, including Zvi, are picturing when you say "legalize blackmail." In fact, de-criminalizing that specific interaction, but leaving alone laws & norms against uncapped extraction, threats, etc. might find few opponents.
Highlights from the Blackmail Debate (Robin Hanson vs Zvi Mowshowitz)

The sensible approach is. to demand a stream of payments over time. If you reveal it to others who also demand streams, that will cut how much of a stream they are willing to pay you.

Highlights from the Blackmail Debate (Robin Hanson vs Zvi Mowshowitz)

You are very much in the minority if you want to abolish norms in general.

3jimmy1yThere's a parallel here with the fifth amendment's protection from self incrimination making it harder to enforce laws and laws being good on average. This isn't paradoxical because the fifth amendment doesn't make it equally difficult to enforce all laws. Actions that harm other people tend to have other ways of leaving evidence that can be used to convict. If you murder someone, the body is proof that someone has been harmed and the DNA in your van points towards you being the culprit. If you steal someone's bike, you don't have to confess in order to be caught with the stolen bike. On the other hand, things that stay in the privacy of your own home with consenting adults are *much* harder to acquire evidence for if you aren't allowed to force people to testify against themselves. They're also much less likely to be things that actually need to be sought out and punished. If it were the case that one coherent agent were picking all the rules with good intent, then it wouldn't make sense to create rules that make enforcement of other rules harder. There isn't one coherent agent picking all the rules and intent isn't always good, so it's important to fight for meta rules that make it selectively hard to enforce any bad rules that get through. You can try to argue that preventing blackmail isn't selective *enough* (or that it selects in the wrong direction), but you can't just equate blackmail with "norm enforcement [applied evenly across the board]".
2Dan B1yI'm not arguing for abolishing norms. You are arguing for dramatically increasing the rate of norm enforcement, and I'm arguing for keeping norm enforcement at the current level. Above, I've provided several examples of ways that I think that increasing the rate of norm enforcement could have bad effects. Do you have some examples of ways that you think that increasing the rate of norm enforcement could have good effects? Note that, for this purpose, we are only counting norm enforcements that are so severe that people would be willing to pay a blackmail fee to escape them. You can't say "there's a norm against littering, so increasing the rate of enforcing that norm would decrease littering" unless you have a plausible scenario in which people would get blackmailed for littering.
Highlights from the Blackmail Debate (Robin Hanson vs Zvi Mowshowitz)

NDAs are also legal in the case where info was known before the agreement. For example, Trump using NDAs to keep affairs secret.

3Richard_Kennaway1yIn that case, a key difference between an NDA and blackmail is that the former fulfils the requirements of a contract, while the latter does not (and not merely by being a currently illegal act). With an NDA where the information is already shared, the party who would prefer that it go no further proactively offers something in return for the other's continued silence. Each party is offering a consideration to the other. If the other party had initiated the matter by threatening to reveal the information unless paid off, there is no contract. Threatening harm and offering to refrain is not a valid consideration. On the contrary, it is the very definition of extortion. Compare cases where it is not information that is at issue. If a housing developer threatens to build an eyesore next to your property unless you pay him off, that is extortion. If you discover that he is planning to build something you would prefer not to be built, you might offer to buy the land from him. That would be a legal agreement. I don't know if you would favour legalising all forms of extortion, but that would be a different argument.
2purge1yBut the typical use of NDAs is notably different from the typical use of blackmail, isn't it? Even though in principle they could be used in all the same situations, they're aren't used that way in practice. Doesn't that make it reasonable to treat them differently?
What can the principal-agent literature tell us about AI risk?

"models are brittle" and "models are limited" ARE the generic complaints I pointed to.

What can the principal-agent literature tell us about AI risk?

We have lots of models that are useful even when the conclusions follow pretty directly. Such as supply and demand. The question is whether such models are useful, not if they are simple.

What can the principal-agent literature tell us about AI risk?

There are THOUSANDS of critiques out there of the form "Economic theory can't be trusted because economic theory analyses make assumptions that can't be proven and are often wrong, and conclusions are often sensitive to assumptions." Really, this is a very standard and generic critique, and of course it is quite wrong, as such a critique can be equally made against any area of theory whatsoever, in any field.

7TAG2yBut of course, it can't be used against them all equally. Physics is so good you can send a probe to a planet millions of miles away. But trying to achieve a practical result in economics is largely guesswork.
6Alexis Carlier2yAside from the arguments we made about modelling unawareness, I don't think we were claiming that econ theory wouldn't be useful. We argue that new agency models could tell us about the levels of rents extracted by AI agents; just that i) we can't infer much from existing models because they model different situations and are brittle, ii) that models won't shed light on phenomena beyond what they are trying to model
What can the principal-agent literature tell us about AI risk?

The agency literature is there to model real agency relations in the world. Those real relations no doubt contain plenty of "unawareness". If models without unawareness were failing to capture and explain a big fraction of real agency problems, there would be plenty of scope for people to try to fill that gap via models that include it. The claim that this couldn't work because such models are limited seems just arbitrary and wrong to me. So either one must claim that AI-related unawareness is of a very different type or scale from ordinary... (read more)

2Tom Davidson2yI agree that the Bostrom/Yudkowsky scenario implies AI-related unawareness is of a very different scale from ordinary human cases. From an outside view perspective, this is a strike against the scenario. However, this deviation from past trends does follow fairly naturally (though not necessarily) from the hypothesis of a sudden and massive intelligence gap
8Alexis Carlier2yThe economists I spoke to seemed to think that in agency unawareness models conclusions follow pretty immediately from the assumptions and so don't teach you much. It's not that they can't model real agency problems, just that you don't learn much from the model. Perhaps if we'd spoken to more economists there would have been more disagreement on this point.
What can the principal-agent literature tell us about AI risk?

"Hanson believes that the principal-agent literature (PAL) provides strong evidence against rents being this high."

I didn't say that. This is what I actually said:

"surely the burden of 'proof' (really argument) should lie on those say this case is radically different from most found in our large and robust agency literatures."

Don't Double-Crux With Suicide Rock

Uh, we are talking about holding people to MUCH higher rationality standards than the ability to parse Phil arguments.

Characterising utopia

"At its worst, there might be pressure to carve out the parts of ourselves that make us human, like Hanson discusses in Age of Em."

To be clear, while some people do claim that such such things might happen in an Age of Em, I'm not one of them. Of course I can't exclude such things in the long run; few things can be excluded in the long run. But that doesn't seem at all likely to me in the short run.

4Richard_Ngo2yApologies for the mischaracterisation. I've changed this to refer to Scott Alexander's post which predicts this pressure.
Don't Double-Crux With Suicide Rock

You are a bit too quick to allow the reader the presumption that they have more algorithmic faith than the other folks they talk to. Yes if you are super rational and they are not, you can ignore them. But how did you come to be confident in that description of the situation?

8jessicata2yBeing able to parse philosophical arguments is evidence of being rational. When you make philosophical arguments, you should think of yourself as only conveying content to those who are rationally parsing things, and conveying only appearance/gloss/style to those who aren't rationally parsing things.

Everything I'm saying is definitely symmetric across persons, even if, as an author, I prefer to phrase it in the second person. (A previous post included a clarifying parenthetical to this effect at the end, but this one did not.)

That is, if someone who trusted your rationality noticed that you seemed visibly unmoved by their strongest arguments, they might think that the lack of agreement implies that they should update towards your position, but another possibility is that their trust has been misplaced! If they find themselves living a world of painted

... (read more)
Another AI Winter?

Seems like you guys might have (or be able to create) a dataset on who makes what kind of forecasts, and who tends to be accurate or hyped re them. Would be great if you could publish some simple stats from such a dataset.

1AnthonyC2yI probably could, but could not share such without permission from marketing, which creates a high risk of bias.
Another AI Winter?

To be clear, Foresight asked each speakers to offer a topic for participants to forecast on, related to our talks. This was the topic I offered. That is NOT the same as my making a prediction on that topic. Instead, that is to say that the chance on this question seemed an unusual combination of verifiable in a year and relevant to the chances on other topics I talked about.

Another AI Winter?

Foresight asked us to offer topics for participants to forecast on, related to our talks. This was the topic I offered. That is NOT the same as my making a prediction on that topic. Instead, that is to say that the chance on this question is an unusual combination of verifiable in a year and relevant to the chances on other topics I talked about.

2Ben Pace2yAh, that makes sense, thanks.

Note that all three of the linked paper are about "boundedly rational agents with perfectly rational principals" or about "equally boundedly rational agents and principals". I have been so far unable to find any papers that follow the described pattern of "boundedly rational principals and perfectly rational agents".

Robin Hanson on the futurist focus on AI

The % of world income that goes to computer hardware & software, and the % of useful tasks that are done by them.

Robin Hanson on the futurist focus on AI

Most models have an agent who is fully rational, but I'm not sure what you mean by "principal is very limited".

Robin Hanson on the futurist focus on AI

I'd also want to know that ratio X for each of the previous booms. There isn't a discrete threshold, because analogies go on a continuum from more to less relevant. An unusually high X would be noteworthy and relevant, but not make prior analogies irrelevant.

Robin, I'm very confused by your response. The question I asked was for references to the specific models you talked about (with boundedly rational principals and perfectly rational agents), not how to find academic papers with the words "principal" and "agent" in them.

Did you misunderstand my question, or is this your way of saying "look it up yourself"? I have searched through the 5 review papers you cited in your blog post for mentions of models of this kind, and also searched on Google Scholar, with negative results. I can try to do more extensive sear

... (read more)
Robin Hanson on the futurist focus on AI

My understanding is that this progress looks much less of a trend deviation when you scale it against the hardware and other resources devoted to these tasks. And of course in any larger area there are always subareas which happen to progress faster. So we have to judge how large is a subarea that is going faster, and is that size unusually large.

Life extension also suffers from the 100,000 fans hype problem.

Robin Hanson on the futurist focus on AI

I'll respond to comments here, at least for a few days.

3Matthew Barnett2yOther than, say looking at our computers and comparing them to insects, what other signposts should we look for, if we want to calibrate progress towards domain-general artificial intelligence?

You previously wrote:

We do have some models of [boundedly] rational principals with perfectly rational agents, and those models don’t display huge added agency rents. If you want to claim that relative intelligence creates large agency problems, you should offer concrete models that show such an effect.

The conclusions of those models seem very counterintuitive to me. I think the most likely explanation is that they make some assumptions that I do not expect to apply to the default scenarios involving humans and AGI. To check this, can you please refere

... (read more)
5ofer2yIt seems you consider previous AI booms to be a useful reference class for today's progress in AI. Suppose we will learn that the fraction of global GDP that currently goes into AI research is at least X times higher than in any previous AI boom. What is roughly the smallest X for which you'll change your mind (i.e. no longer consider previous AI booms to be a useful reference class for today's progress in AI)? [EDIT: added "at least"]
1NaiveTortoise2yMostly unrelated to your point about AI, your comments about the 100,000 fans having the potential to cause harm rang true to me. Are there other areas in which you think the many non-expert fans problem is especially bad (as opposed to computer security, which you view as healthy in this respect)? -------------------------------------------------------------------------------- Would you consider progress on image recognition and machine translation as outside view evidence for lumpiness? Accuracies on ImageNet, an image classification benchmark, dropped by >10% over a 4-year period (graph below) mostly due to the successful scaling up of a type of neural network. This also seems relevant to your point about AI researchers who have been in the field for a long time being more skeptical. My understanding is that most AI researchers would not have predicted such rapid progress on this benchmark before it happened. That said, I can see how you still might argue this is an example of over-emphasizing a simple form of perception, which in reality is much more complicated and involves a bunch of different interlocking pieces.
Prediction Markets Don't Reveal The Territory

Markets can work fine with only a few participants. But they do need sufficient incentives to participate.

Prediction Markets Don't Reveal The Territory

"of all the hidden factors which caused the market consensus to reach this point, which, if any of them, do we have any power to affect?" A prediction market can only answer the question you ask it. You can use a conditional market to ask if a particular factor has an effect on an outcome. Yes of course it will cost more to ask more questions. If there were a lot of possible factors, you might offer a prize to whomever proposes a factor that turns out to have a big effect. Yes it would cost to offer such a prize, because it could be work to find such factors.

1JohnBuridan2yGood point. But it is not just a cost problem. My conjecture in the above comment is that conditional markets are more prone to market failure because the structure of conditional questions decreases the pool of people who can participate. I need more examples of conditional markets in action to figure out what the greatest causes of market failure are for conditional markets.
Quotes from Moral Mazes

I was once that young and naive. But I'd never heard of this book Moral Mazes. Seems great, and I intend to read it.

Simple Rules of Law

The CEO proposal is to fire them at the end of the quarter if the prices just before then so indicate. This solves the problem of the market traders expecting later traders to have more info than they. And it doesn't mean that the board can't fire them at other times for other reasons.

We could expect prices prior to end of quarter to be strange, then, and potentially containing very strange information, but can also argue it shouldn't matter. So this is like the two-stage proposal. In stage 1 board decides whether to fire or not anyway, in stage 2 the prediction market decides whether to fire him anyway with a burst of activity, which has the advantage that you get your money back fast if it doesn't happen, and if it does happen you can just be long/short the stock. then if the board decides to fire him because it was 'to... (read more)

Strategic implications of AIs' ability to coordinate at low cost, for example by merging

The claim that AI is vastly better at coordination seems to me implausible on its face. I'm open to argument, but will remain skeptical until I hear good arguments.

1MakoYass3yI'd expect a designed thing to have much cleaner, much more comprehensible internals. If you gave a human a compromise utility function and told them that it was a perfect average of their desires (or their tribe's desires) and their opponents' desires, they would not be able to verify this, they wouldn't recognise their utility function, they might not even individually possess it (again, human values seem to be a bit distributed), and they would be inclined to reject a fair deal, humans tend to see their other only in extreme shades, more foreign than they really are. Do you not believe that an AGI is likely to be self-comprehending? I wonder, sir, do you still not anticipate foom? Is it connected to that disagreement?

Have you considered the specific mechanism that I proposed, and if so what do you find implausible about it? (If not, see this longer post or this shorter comment.)

I did manage to find a quote from you that perhaps explains most of our disagreement on this specific mechanism:

There are many other factors that influence coordination, after all; even perfect value matching is consistent with quite poor coordination.

Can you elaborate on what these other factors are? It seems to me that most coordination costs in the real world come from value differences,

... (read more)
2Dagon3yAs a subset of the claim that AI is vastly better at everything, being vastly better at coordination is plausible. The specific arguments that AI somehow has (unlike any intelligence or optimization process we know of today) introspection into it's "utility function" or can provide non-behavioral evidence of it's intent to similarly-powerful AIs seem pretty weak. I haven't seen anyone attempting to model shifting equilibria and negotiation/conflict among AIs (and coalitions of AIs and of AIs + humans) with differing goals and levels of computational power, so it seems pretty unfounded to speculate on how "coordination" as a general topic will play out.
5ryan_b3yIt seems to me that computers don't suffer from most of the constraints humans do. For example, AI can expose its source code and its error-less memory. Humans have no such option, and our very best approximations are made of stories and error-prone memory. They can provide guarantees which humans cannot, simulate one another within precise boundaries in a way humans cannot, calculate risk and confidence levels in a way humans cannot, communicate their preferences precisely in a way humans cannot. All of this seems to point in the direction of increased clarity and accuracy of trust. On the other hand, I see no reason to believe AI will have the strong bias in favor of coordination or trust that we have, so it is possible that clear and accurate trust levels will make coordination a rare event. That seems off to me though, because it feels like saying they would be better off working alone in a world filled with potential competitors. That statement flatly disagrees with my reading of history.
4habryka3yWhat evidence would convince you otherwise? Would superhuman performance in games that require difficult coordination be compelling? Deepmind has outlined Hanabi as one of the next games to tackle: []
Robin Hanson on Simple, Evidence Backed Models

Secrecy CAN have private value. But it isn't at all clear that we are typically together better off with secrets. There are some cases, to be sure, where that is true. But there are also so many cases where it is not.

6cousin_it3yIt seems to me that removing privacy would mostly help religions, political movements and other movements that feed on conformity of their members. That doesn't seem like a small thing - I'm not sure what benefit could counterbalance that.
2Dagon3yQuite agree - depending on how you aggregate individual values and weigh the adversarial motives, it's quite possible that "we" are often worse off with secrets. It's not clear whether or when that's the case from the "simple model" argument, though. And certainly there are cases where unilateral revelations while others retain privacy are harmful. Anytime you'd like to play poker where your cards are face-up and mine are known only to me, let me know. I would love to explore whether private information is similar to other capital, where overall welfare can be improved by redistribution, but only under certain assumptions of growth, aggregation and individual benefits canceling out others' harms.
Why didn't Agoric Computing become popular?

My guess is that the reason is close to why security is so bad: Its hard to add security to an architecture that didn't consider it up front, and most projects are in too much of a rush to take time to do that. Similarly, it takes time to think about what parts of a system should own what and be trusted to judge what.. Easier/faster to just make a system that does things, without attending to this, even if that is very costly in the long run. When the long run arrives, the earlier players are usually gone.

Some disjunctive reasons for urgency on AI risk

We have to imagine that we have some influence over the allocation of something, or there's nothing to debate here. Call it "resources" or "talent" or whatever, if there's nothing to move, there's nothing to discuss.

I'm skeptical solving hard philosophical problems will be of much use here. Once we see the actual form of relevant systems then we can do lots of useful work on concrete variations.

I'd call "human labor being obsolete within 10 years … 15%, and within 20 years … 35%" crazy extreme predi... (read more)

8Wei_Dai3yLet me rephrase my argument to be clearer. You suggested earlier, "and if resources today can be traded for a lot more resources later, the temptation to wait should be strong." This advice could be directed at either funders or researchers (or both). It doesn't seem to make sense for researchers, since they can't, by not working on AI alignment today, cause more AI alignment researchers to appear in the future. And I think a funder should think, "There will be plenty of funding for AI alignment research in the future when there are clearer warning signs. I could save and invest this money, and spend it in the future on alignment, but it will just be adding to the future pool of funding, and the marginal utility will be pretty low because at the margins, it will be hard to turn money into qualified alignment researchers in the future just as it is hard to do that today." So I'm saying this particular reallocation of resources that you suggested does not make sense, but the money/talent could still be reallocated some other way (for example to some other altruistic cause today). Do you have either a counterargument or another suggestion that you think is better than spending on AI alignment today? Have you seen my recent posts that argued for or supported this? If not I can link them: Three AI Safety Related Ideas [] , Two Neglected Problems in Human-AI Safety [] , Beyond Astronomical Waste [], The Argument from Philosophical Difficulty [] . Sure, but why can't philosophical work be a complement to that? I won't defend these numbers because I haven't put much thought into this topic personally (since my own reasons don't depend on
Some disjunctive reasons for urgency on AI risk

Solving problems is mostly a matter of total resources devoted, not time devoted. Yes some problems have intrinsic clocks, but this doesn't look like such a problem. If we get signs of a problem looming, and can devote a lot of resources then, that makes it tempting to save resources today for such a future push, as we'll know a lot more then and resources today become more resources when delayed.

In software development, a perhaps relevant kind of problem solving, extra resources in the form of more programmers working on the same project doesn't speed things up much. My guesstimate is output = time x log programmers. I assume the main reason being because there's a limit to the extent that you can divide a project into independent parallel programming tasks. (Cf 9 women can't make a baby in 1 month.)

Except that if the people are working in independent smaller teams, each trying to crack the same problem, and *if* the solution requir... (read more)

I think that this problem is in the same broad category as "invent general relativity" or "prove the Poincare conjecture". That is, for one thing quantity doesn't easily replace talent (you couldn't invent GR just as easily with 50 mediocre physicists instead of one Einstein), and, for another thing, the work is often hard to parallelize (50 Einsteins wouldn't invent GR 50 times as fast). So, you can't solve it just by spending lots of resources in a short time frame.

Solving problems is mostly a matter of total resources devoted, not time devoted. ... If we get signs of a problem looming, and can devote a lot of resources then.

Hmm. I don't have as strong opinions about this, but this premise doesn't seem obviously true.

I'm thinking about the "is science slowing down?" question – pouring 1000x resources into various scientific fields didn't result in 1000x speedups. In some cases progress seemed to slow down. The three main hypotheses I have are:

  • Low hanging fruit got used up, so the problem
... (read more)
Some disjunctive reasons for urgency on AI risk

Can you point to a good/best argument for the claim that AGI is coming soon enough to justify lots of effort today?

9Wei_Dai3yI'm not actually aware of a really good argument for AGI coming soon (i.e., within next few decades). As far as I can tell, most people use their own intuitions and/or surveys of AI researchers (both of which are of course likely to be biased). My sense is that it's hard to reason explicitly about AGI timelines (in a way that's good enough to be more trustworthy than intuitions/surveys) and there seem to be enough people concerned about foom and/or short timelines that funding isn't a big constraint so there's not a lot of incentives for AI risk people to spend time on making such explicit arguments. (ETA: Although I could well be wrong about this, and there's a good argument somewhere that I'm not aware of.) To give a sense of how people are thinking about this, I'll quote a Paul Christiano interview [] : My own thinking here is that even if AGI comes a century or more from now, the safest alignment approaches [] seem to require solving a number of hard philosophical problems which may well take that long to solve even if we start now. Certainly it would be pretty hopeless if we only started when we saw a clear 10-year warning. This possibility also justifies looking more deeply into other approaches now to see if they could potentially be just as safe without solving the hard philosophical problems. Another thought that is prompted by your question is that given funding does not seem to be the main constraint on current alignment work (people more often cite "talent"), it's not likely to be a limiting constraint in the future either, when the warning signs are even clearer. But "resources today can be traded for a lot more resources later" doesn't seem to apply if we interpret "resources" as "talent".
Some disjunctive reasons for urgency on AI risk

Its not quite about "fast" v. "slow" than about the chances for putting lots of resources into the problem with substantial warning. Even if things change fast, as long as you get enough warning and resources can be moved to the problem fast enough, waiting still makes sense.

Some disjunctive reasons for urgency on AI risk

By "main reason for concern" I mean best arguments; I'm not trying to categorize people's motivations.

AGI isn't remotely close, and I just don't believe people who think they see signs of that. Yes for any problem that we'll eventually want to work on, a few people should work on it now just so someone is tracking the problem, ready to tell the rest of us if they see signs of it coming soon. But I see people calling for much more than that minimal tracking effort.

Most people who work in research areas call for more rela... (read more)

AGI isn’t remotely close, and I just don’t believe people who think they see signs of that.

You don't seem to believe in foom either, but you're at least willing to mention it as a reason some people give for urgency and even engage in extensive debates about it. I don't understand how "no foom, but AGI may be close enough that it's worthwhile to do substantial alignment work now" could be so much less likely in your mind than foom that it's not even worth mentioning as a reason that some other (seemingly smart and sane) people give for urgency.

Most pe

... (read more)
Some disjunctive reasons for urgency on AI risk

Even if there will be problems worth working on at some point, if we will know a lot more later and if resources today can be traded for a lot more resources later, the temptation to wait should be strong. The foom scenario has few visible indications of a problem looming, forcing one to work on the problems far ahead of time. But in scenarios where there's warning, lots more resources, and better tools and understand later, waiting makes a lot more sense.

4Gurkenglas3yIf there is a 50-50 chance of foom vs non-foom, and in the non-foom scenario we expect to acquire enough evidence to get an order of magnitude more funding, then to maximize the chance of a good outcome we, today, should invest in the foom scenario because the non-foom scenario can be handled by more reluctant funds.

Conditional on the nonfoom scenario, what is the appropriate indication that you should notice, to start converting resources into work?

If the world where there may or may not be a foom, how likely does foom need to be to make it correct to work on sooner?

If you agree that there will be problems worth working on at some point, then when to start working on them becomes a judgement call about how hard the problems are, which warning sign will leave enough time to solve them, how much better tools and understanding will get in the future (without us working specifically to improve such tools/understanding), and how current resources trade against future resources. If you agree with this, I suggest that another reason for urgency besides foom is a judgment that we've already passed such warning signs where it

... (read more)
1ioannes3yRelated [] , on the EA Forum. (I am the post's author.)
Towards no-math, graphical instructions for prediction markets

Your critique is plausible. I was never a fan of these supposedly simple interfaces.

Towards no-math, graphical instructions for prediction markets

There have long been lots of unexplored good ideas for interface improvements. But they need to be tested in the context of real systems and users.

On Robin Hanson’s Board Game

Yes, we only did a half dozen trials, and mostly with new players, so players were inexperienced.

On Robin Hanson’s Board Game

Note that all this analysis is based on thinking about the game, not from playing the game. From my observing game play, I'd say that price accuracy does not noticeably suffer in the endgame.

For game design, yes good to include a few characters who will be excluded early, so people attend to story in early period.

What types of players did you test the game on, and how many games did they each play?

I can think of many other games where this distortion effect doesn't happen with new players, as they don't think about the game ending or the strategic layer, then picks up as players gain experience and improve. So this result isn't that surprising for players on their first game, especially if they're not hardcore game players. But it would be surprising if it was a stable equilibrium.

On Robin Hanson’s Board Game

If more than one person "did it", you could pay off that fraction of $100 to each. So if two did it, each card is worth $50 at the end.

2Zvi3yI like it, that works well, so long as we have an airtight definition in advance of when this counts. Alternatively, we can know from our guide that the result won't be ambiguous.
1ParanoidAltoid3yThat works. Though now card price doesn't actually reflect a character's implied probability of surviving. Eg buying a card at $40 is a confident move if there's a chance of two survivors, and always loses money if there's 3. Instead it'd be $100*p(surviving|1 survivor) + $50*p(surviving|2 survivors) + $33*p(surviving|3 survivors)... which makes it a lot harder to think about whether to buy or sell. Could make things more interesting though.
Confusions Concerning Pre-Rationality

The problem of how to be rational is hard enough that one shouldn’t expect to get good proposals for complete algorithms for how to be rational in all situations. Instead we must chip away at the problem. And one way to do that is to slowly collect rationality constraints. I saw myself as contributing by possibly adding a new one. I’m not very moved by the complaint “but what is the algorithm to become fully rational from any starting point?” as that is just too hard a problem to solve all at once. 

I don't think "what is the algorithm to become fully rational from any starting point?" is a very good characterization. It is not possible to say anything of interest for any starting point whatsoever. I read Wei Dai as instead asking about the example he provided, where a robot is fully rational in the standard Bayesian sense (by which I mean, violates no laws of probability theory or of expected utility theory), but not pre-rational. It is then interesting to ask whether we can motivate such an agent to be pre-rational, or, failing that, ... (read more)

What Evidence Is AlphaGo Zero Re AGI Complexity?

I disagree with the claim that "this single simple tool gives a bigger advantage on a wider range of tasks than we have seen with previous tools."

Load More