Sorted by New


Should we postpone AGI until we reach safety?

You say 'I've been in or near this debate since the 1990s'. That suggests there are many people with my opinion. Who?

Nick Bostrom comes to mind as at least having a similar approach. And it's not like he's without allies, even in places like Less Wrong.

... and, Jeez, back when I was paying more attention, it seemed like some kind of regulation, or at least some kind of organized restriction, was the first thing a lot of people would suggest when they learned about the risks. Especially people who weren't "into" the technology itself.

I was hanging around the Foresight Institute. People in that orbit were split about 50-50 between worrying most about AI and worrying most about nanotech... but the two issues weren't all that different when it came to broad precautionary strategies. The prevailing theory was roughly that the two came as a package anyway; if you got hardcore AI, it would invent nanotech, and if you got nanotech, it would give you enough computing power to brute-force AI. Sometimes "nanotech" was even taken as shorthand for "AI, nanotech, and anything else that could get really hairy"... vaguely what people would now follow Bostrom and call "X-risk". So you might find some kindred spirits by looking in old "nanotech" discussions.

There always seemed to be plenty of people who'd take various regulate-and-delay positions in bull sessions like this one, both online and offline, with differing degrees of consistency or commitment. I can't remember names; it's been ages.

The whole "outside" world also seemed very pro-regulation. It felt like about every 15 minutes, you'd see an op-ed in the "outside world" press, or even a book, advocating for a "simple precautionary approach", where "we" would hold off either as you propose, until some safety criteria were met, or even permanently. There were, and I think still are, people who think you can just permanently outlaw something like AGI ,and that will somehow actually make it never happen. This really scared me.

I think the word "relinquishment" came from Bill McKibben, who I as I recall was, and for all I know may still be, a permanent relinquishist, at least for nanotech. Somebody else had a small organization and phrased things in terms of the "precautionary principle". I don't remember who that was. I do remember that their particular formulation of the precautionary principle was really sloppy and would have amounted to nobody ever being allowed to do anything at all under any circumstances.

There were, of course, plenty of serious risk-ignorers and risk-glosser-overs in that Foresight community. They probably dominated in many ways, even though Foresight itself definitely had a major anti-risk mission component. For example, an early, less debugged version of Eliezer Yudkowsky was around. I think, at least when I first showed up, he still held just-blast-ahead opinions that he has, shall we say, repudiated rather strongly nowadays. Even then, though, he was cautious and level-headed compared to a lot of the people you'd run into. I don't want to make it sound like everybody was trying to stomp on the brakes or even touch the brakes.

The most precautionary types in that community probably felt pretty beleaguered, and the most "regulatory" types even more so. But you could definitely still find regulation proponents, even among the formal leadership.

However, it still seems to me that ideas vaguely like yours, while not uncommon, were often "loyal opposition", or brought in by newcomers... or they were things you might hear from the "barbarians at the gates". A lot of them seemed to come from environmentalist discourse. On the bull-session level, I remember spending days arguing about it on some Greenpeace forum.

So maybe your problem is that your personal "bubble" is more anti-regulation than you are? I mean, you're hanging out on Less Wrong, and people on Less Wrong, like the people around Foresight, definitely tend to have certain viewpoints... including a general pro-technology bias, an urge to shake up the world, and often extremely, even dogmatically anti-regulation political views. If you looked outside, you might find more people who think they way you do. You could look at environmentalism generally, or even at "mainstream" politics.

Should we postpone AGI until we reach safety?

Not to speak for Dagon, but I think point 2 as you write it is way, way too narrow and optimistic. Saying "it would be rather difficult to get useful regulation" is sort of like saying "it would be rather difficult to invent time travel".

I mean, yes, it would be incredibly hard, way beyond "rather difficult", and maybe into "flat-out impossible", to get any given government to put useful regulations in place... assuming anybody could present a workable approach to begin with.

It's not a matter of going to a government and making an argument. For one thing, a government isn't really a unitary thing. You go to some *part *of a government, fight to even get your issue noticed. Then you compete with all the other people who have opinions. Some of them will directly oppose your objectives. Others will suggest different approaches, leading to delays in hashing out those differences, and possibly to compromises that are far less effective than any of the sides' "pure" proposals.

Then you get to take whatever you hashed out with the part of the government you've started dealing with, and sell it in all of the other parts of that government and the people who've been lobbying them. In the process, you find out about a lot of oxen you propose to gore that you didn't even imagine existed.

In big countries, people often spend whole careers in politics, arguing, fighting, building relationships, doing deals... to get even compromised, watered-down versions of the policies they came in looking for.

But that's just the start. You have to get many governments, possibly almost all governments, to put in similar or at least compatible regulations... bearing in mind that they don't trust each other, and are often trying either to compete with each other, or to advantage their citizens in competing with each other. Even that formulation is badly oversimplified, because governments aren't the only players.

You also have to get them to apply those regulations to themselves, which is hard because they will basically all believe that the other governments are cheating, and probably that the private sector is also cheating... and they will probably be right about that. And of course it's very easy for any kind of leader to kid themselves that their experts are too smart to blow it, whereas the other guys will probably destroy the world if they get there first.

Which brings you to compliance, whether voluntary or coerced, inside and outside of governments. People break laws and regulations all the time. It's relatively easy to enforce compliance if what you're trying to stamp out is necessarily large-scale and conspicuous... but not all dangerous AI activity necessarily has to be that way. And nowadays you can coordinate a pretty large project in a way that's awfully hard to shut down.

Then there's the blowback. There's a risk of provoking arms races. If there are restrictions, players have incentives to move faster if they think the other players are cheating and getting ahead... but they also have incentives to move if they think the other players are not cheating ,and can therefore be attacked and dominated. If a lot of the work is driven into secrecy, or even if people just think there might be secret work, then there are lots of chances for people to think both of those things... with uncertainty to make them nervous.

... and, by the way, by creating secrecy, you've reduced the chance of somebody saying "Ahem, old chaps, have you happened to notice that this seemingly innocuous part of your plan will destroy the world"? Of course, the more risk-averse players may think of things like that themselves, but that just means that the least risk-averse players become more likely first movers. Probably not what you wanted.

Meanwhile, resources you could be using to win hearts and minds, or to come up with technical approaches, end up tied up arguing for regulation, enforcing regulation, and complying with regulation.

... and the substance of the rules isn't easy, either. Even getting a rough, vague consensus on what's "safe enough" would be really hard, especially if the consensus had to be close enough to "right" to actually be useful. And you might not be able to make much progress on safety without simultaneously getting closer to AGI. For that matter, you may not be able to define "AGI" as well as you might like... nor know when you're about to create it by accident, perhaps as a side effect of your safety research. So it's not as simple as "We won't do this until we know how to do it safely". How can you formulate rules to deal with that?

I don't mean to say that laws or regulations have no place, and still less do I mean to say that not-doing-bloody-stupid-things has no place. They do have a place.

But it's very easy, and very seductive, to oversimplify the problem, and think of regulation as a magic wand. It's nice to dream that you can just pass a law, and this or that will go away, but you don't often get that lucky.

"Relinquish this until it's safe" is a nice slogan, but hard to actually pin down into a real, implementable set of rules. Still more seductive, and probably more dangerous, is the idea that, once you do come up with some optimal set of rules, there's actually some "we" out there that can easily adopt them, or effectively enforce them. You can do that with some rules in some circumstances, but you can't do it with just any rules under just any circumstances. And complete relinquishment is probably not one you can do.

In fact, I've been in or near this particular debate since the 1990s, and I have found that the question "Should we do X" is a pretty reliable danger flag. Phrasing things that way invites the mind to think of the whole world, or at least some mythical set of "good guys", as some kind of unit with a single will, and that's just not how people work. There is no "we" or "us", so it's dangerous to think about "us" doing anything. It can be dangerous to talk about any large entity, even a government or corporation, as though it had a coordinated will... and still more so for an undefined "we".

The word "safe" is also a scary word.

Legalize Blackmail: An Example

It would especially be a waste of time to copy and paste Hanson's stuff because, "Checkmate" subject lines aside, as far as I can tell, he's never posted anything that addressed anything I've said in this thread at all. And he keeps repeating himself while failing to engage properly with important objections... including clearly consequentialist ones.

We were talking about a case where the blackmailable behavior was already going on. As far as I can tell, Hanson hasn't mentioned the question of whether blackmail provides any meaningful incentive to stop an existing course of blackmailable behavior. I don't know what he'd say. I don't think it does.

If you want to broaden the issue to blackmail in general, Hanson's basic argument seems to be that having to pay blackmail is approximately as much a punishment for bad behavior as gossip would be, while the availability of cash payments would attract significantly more people who could actually administer such punishment. Therefore, if you want to see some particular behavior punished, you should permit blackmail regarding at least that kind of behavior.

He apparently takes it as given that blackmail being more available would meaningfully increase disincentives for "blackmailable" behavior, and would therefore actually reduce such behavior enough to be interesting. That's where he starts, and everything from then on is argued on that assumption. I don't see anywhere where he justifies it. He appears to think it's obvious.

I don't think it's true or even plausible.

I think Hanson probably overestimates how much legalized blackmail (and presumably more socially acceptable blackmail) would enlarge the pool of potential "enforcers". Blackmail is already somewhat hard to punish, so you'd expect formal legalization to have limited effect on its attractiveness. On the "supply side", it's much easier to act opportunistically on compromising information than it is to go into the business of speculatively trying to develop compromising information on specific targets.

But he could be right about that part, assuming a rational blackmailer. Legality has a big effect on large-scale, organized group activity. Maybe if it were legal, you really would have more organized, systematic attempts to investigate high-value targets with the idea of blackmailing them.

Where I think he's most surely wrong is on the strength of the effect that would have on the behavior of the potential blackmail-ee. I don't think that, in practice, significantly fewer people would choose to initiate blackmailable behavior.

Law, gossip, and illegal blackmail already produce most of the disincentives that legalized blackmail would. You already run a pretty large risk of anything you do being exposed. Much more importantly, it doesn't matter, because people don't react rationally or proportionately to that kind of disincentive.

People do not in fact think "I won't do this because I might be blackmailed". At the most, they may think "I won't do this because I might be found out", but they don't then break down the consequences of being found out into being blackmailed, being gossiped about, or just being thought badly of. Even noticing that something being found out might be an issue is more than most people will do. And forget about anybody consciously assigning an equivalent monetary value to being found out.

People might, maybe, think separately about the consequences of being punished criminally, but honestly I think the psychological mechanism there is much more "I don't want to see myself as a criminal", than "I don't want to get punished". People who actually commit crimes overwhelmingly don't expect to get caught, whether or not that expectation is rational. No plausible amount of blackmail-oriented investigation is going to change that expectation, any more than the already fairly large amount of government criminal investigation does.

Hanson seems to want to talk about a world of logically omniscient, rational , consequentialist actors, with unlimited resources to spend on working out the results of every alternative, and the ability to put a firm monetary value on everything. On a lot of issues, the vast majority of humans are not even approximately logically omniscient, rational, consequentialist actors, and the monetary values they assign to things are deeply inconsistent. Deviant behavior is one of the areas where people are least rational. Blackmail is all about deviant behavior, so Hanson's whole analytical framework is inapproprate from the get-go. It's probably a bit less unreasonable for modeling the conduct of blackmailers, but it's wholly wrong for modeling the behavior of blackmail-ees.

Speaking of that "rationality differential", there was also a suggestion in comments that legalized blackmail would give potential blackmailers an incentive to induce blackmailable behavior. That's pretty plausible and actually happens. It's 101 material in spy school, for example. It's often done by intentionally exploiting flaws in the target's rationality.

As far as I could see, Hanson just ignored that part of the comment. I'm tempted to put words into his mouth and imagine him saying "Why would you let a blackmailer induce you into blackmailable behavior? That's stupid.". Well, maybe it is stupid, but it happens all the time. So we have at least one unambiguously negative effect that he's totally ignored. All by itself, that one negative effect would probably create more blackmailable behavior than fear of blackmail would deter.

Hanson doesn't even concede enough to address the fact that the risk of blackmail is more manageable, and therefore possibly more appealing, than the risk of actual exposure, so a somewhat-less-extremely-unrealistic human actor might actually be more inclined to engage in "blackmailable" behavior if they expected anyone who happened to discover their behavior to blackmail them, rather than to simply expose them.

He's also talking about changes to law, but he ignores all of the factors that make legal systems gameable and load those systems up with friction. Unless forced, he more or less models the legal system as purely mechanical, and doesn't care to get into how actors actually use and abuse legal processes, nor systematic differences in various actors' sophistication, skills or resources for doing so. When negative effects based in those areas are brought up in comments, he just doubles down and suggests creating ever larger, more sweeping, systemwide legal changes essentially by fiat... which is politically impossible. He might as well suggest changing the law of gravity.

The behavior of homo economicus in a frictionless environment is simply uninteresting, and isn't quite as cut-and-dried as Hanson would have it anyway. The observed behavior of real humans suggests significant negative effects from putting his proposal into actual practice, and argues against the idea than any of his suggested benefits would be realized. And suggesting policy changes that are politically impossible is largely pointless anyway.

Personally, I'm not sure legalized blackmail would make a big difference in the world as a whole, but, if it did, I would expect it to be far more negative than positive. I would expect it to create a world with more snares, more fear, much more deliberately induced bad behavior, significantly more system gaming of all kinds, while not deterring much if any pre-existing bad behavior. Hanson hasn't said anything that changes that impression.

I will now go back to my usual policy of ignoring Hanson completely...

Legalize Blackmail: An Example

SM only had to have Zahara arrested to tarnish his personal reputation and prevent whistleblowing. Future whistleblowers can see what happened to Zahara and will choose not to come forward.

Well, OK, but in a counterfactual world with legal blackmail, other failure modes would surely show up. For example, if blackmail were legal, a target would have an easier time casting doubt on a whistleblower by claiming that the whole accusation was made up for financial reasons. Or finding something legitimately embarrassing and, well, blackmailing the whistleblower into silence. All completely legal, and much easier for a more powerful party to pull off than for a less powerful party.

For that matter, you could see other ways to chill legal process specifically. I have no idea what grounds this guy sued on to begin with, but I doubt it was airtight. You might get legal advice like "You have an iffy case suing over this, so why don't you just blackmail them?". You could even imagine a court saying "Well, you don't have standing because you didn't take a reasonably available option to remedy the harm to you, namely blackmailing the defendant".

... and if you just want to frame somebody for any old crime to discredit them, it's easy to substitute something else for blackmail. Very possibly even something else directly related to the case at hand.

Treating this case as an argument for legal blackmail seems like a weird cherry-picked argument that relies on very specific and unusual facts.

We do know that, if blackmail were legal, we would have better information about which world we are in.

Maybe in this case, but if blackmail were legal, there would, in general, be an incentive to monetize negative information about various actors, rather than publicizing it. How does that lead to more information?

So far as I can tell, if blackmail were legal...

  1. There would be some tiny additional risk to targets beyond their already large risk of being exposed. Very likely not enough risk to deter a significant amount of malfeasance before the fact.

  2. Financial incentives would convert some whistleblowers into blackmailers. Whereas a whistleblower may force some bad activity to stop, a blackmailer is really only guaranteed to collect money because of their knowledge.

I want to emphasize that... I don't see where blackmail would give ever anybody any incentive to actually stop any malfeasance.

A halfway competent blackmailer will make the demand more attractive (to the decision maker) than the consequences of exposure. Assuming the blackmailer could convince the target that the demand was supportable and that no insupportable future demands would be made, you'd expect the target to pay. And in a world with legal blackmail, the target would presumably have legal process available to keep the blackmailer from making extra demands later, so that "trust" would be easier to get (this system is really starting to look like a paradise for crazy legal games).

Once having paid, and thereby eliminated the immediate risk of exposure, the target would have no reason not to simply keep going with the bad behavior. If anything, the target might be hungrier for cash to pay ongoing blackmail.

The target's risk analysis after the blackmail is pretty much the same as it was before the blackmail. Obviously the target started the malfeasance in the first place, and has continued it thereafter, so that analysis must favor the malfeasance.

Of course, the blackmailer could make stopping part of the demand... but that demands an altruistic choice to demand "pay me and stop" rather than "pay me more". Even then, it only works if the target believes that the blackmailer can and will actually find out about any failure to stop.

Postmortem to Petrov Day, 2020

Would it also be reasonable for a user to expect that the administrator of a site would not expose it to being shut down by some random person, if the administrator did not see the matter as a game?

Postmortem to Petrov Day, 2020

Indeed, last year I know of a user (not Chris Leong) who visited the site, clicked the red button, and entered and submitted codes, before finding out what the button did.

As a result of that user, this year we changed the red button to the following, so that mistake would not happen again.

If I showed up cold (not as a person who'd actually been issued codes and not having advance knowledge of the event), and saw even the 2020 button with "Click here to destroy Less Wrong", it would never cross my mind that clicking it would actually have any effect on the site, regardless of what it says.

I'd assume it was some kind of joke or an opportunity to play some kind of game. My response would be to ignore it as a waste of time... but if I did click on it for some reason, and was asked for a code, I'd probably type some random garbage to see what would happen. Still with zero expectation of actually affecting the site.

Who would believe what it says? It's not even actually true; "destroy" doesn't mean "shut down for a day".

The Web is full of jokey "red buttons", and used to have more, so the obvious assumption is that any one you see is just another one of those.

AGI safety from first principles: Control

Why would you expect it to be "us" versus "the AI" (or "the AIs")? Where's this "us" coming from?

I would think it would be very, very likely for humans to try to enlist AGIs as allies in their conflicts with other humans, to rush the development and deployment of such AGIs, to accept otherwise unacceptable risks of mistakes, to intentionally remove constraints they'd otherwise put on the AGIs' actions, and to give the AGIs more resources than they'd otherwise get. It's not just that you can't rely on a high level of coordination; it's that you can rely on a high level of active conflict.

There'll always be the assumption that if you don't do it first, the other guy will do it to you. And you may rightly perceive the other guy as less aligned with you than the AGI is, even if the AGI is not perfectly aligned with you either.

Of course, you could be wrong about that, too, in which case the AGI can let the two of you fight, and then mop up the survivors. Probably using the strategy and tactic module you installed.

Hiring engineers and researchers to help align GPT-3

it would be much better if we had an API that was trying to help the user instead of trying to predict the next word of text from the internet.

"I'm from OpenAI, and I'm here to help you".

Seriously, it's not obvious that you're going to do anything but make things worse by trying to make the thing "try to help". I don't even see how you could define or encode anything meaningfully related to "helping" at this stage anyway.

As for the bottom line, I can imagine myself buying access to the best possible text predictor, but I can't imagine myself buying access to something that had been muddied with whatever idea of "helpfulness" you might have. I just don't want you or your code making that sort of decision for me, thanks.

What should I teach to my future daughter?
Answer by jbashJun 19, 202012

I suggest that you relax a bit. She's not going to be learning programming or anything like it for years, regardless. Newborns spend months to years just learning how to use their own limbs and process the information from their own senses.

And I've never heard any evidence at all that, say, programming, is particularly important to learn all that early in life. Manual/mental skills like musical performance seem to turn out best if started early (but not necessarily as a toddler!). Languages, too. I could even imagine that critical logical thinking would benefit from early exposure. But programming? That's something you can figure out.

In the long run, meta-skills are important... things that let you decide for yourself which skills to learn and learn them on your own. And things that let you evaluate both the truth and the usefulness of all the stuff that everybody else is trying to teach you. Beyond that, the more flexible and generalizable the better.

But the biggest thing is this: she's going to be her own person. By the time she's old enough to be taught the kinds of hands-on skills you're talking about, she's going to have her own ideas about what she wants to learn. "This civilization" isn't some kind of apocalyptic dystopia, and you don't know "what is coming". In all probability, it will all add up to normality. In all probability, she will muddle through. ... and in all probability, neither you nor anybody here can guess what very specific skills she's going to need. Assuming, that is, that human skills are even relevant at all when she grows up.

Please don't drive her insane by pushing "needed practical skills". Let her enjoy life, and let her learn by doing things that engage her. While you're unlikely to make a monster impact by predicting what she'll need in the future, you will definitely have an impact on her present, and maybe on how she sees learning in general..

On “COVID-19 Superspreader Events in 28 Countries: Critical Patterns and Lessons”

Um, direction of airflow, by definition, doesn't affect the ballistic transmission of anything. On the other hand, the longer something hangs in the air, the more it's affected by the direction of airflow, and that applies all the way down to gas molecules.

Singing or breathing hard seems likely to increase droplets of all sizes right down to submicron.

Load More