Convincing All Capability Researchers

Logan Riggs

120 Convincing All Capability Researchers

8th Apr 2022

4 min read

120

This post is heavily based off this excellent comment (the author's name is not relevant)

Give the world's thousand most respected AI researchers $1M each to spend 3 months working on AI alignment, with an extra $100M if by the end they can propose a solution alignment researchers can't shoot down. I promise you that other than like 20 industry researchers who are paid silly amounts, every one of them would take the million. They probably won't make any progress, but from then on when others ask them whether they think alignment is a real unsolved problem, they will be way more likely to say yes. That only costs you a billion dollars! I literally think I could get someone reading this the money to do this (at least at an initially moderate scale) - all it needs is a competent person to step up.

Usually people object to paying people large sums of money to work on alignment because they don't expect them to produce any good work (mostly because it's very hard to specify alignment, see below). This is a feature, not a bug.

Being able to say "the 1000 smartest people working in AI couldn't make headway in Alignment in 3 months even when they were paid $1 million and a solution would be awarded $100 million" is very good for persuading existing researchers that this is a very hard problem.

Would this really go a long way to convincing those at major AI labs that alignment is hard? We could actually ask people working at these places if there were no progress made in those 3 months if it would change their minds.

Another problem is looooong timelines. They can agree it's hard, but their timelines may be too far to matter to work on it now. I could think of a couple counter-arguments that may work (but it would better to actually test them with real people)

1. To predict AGI (or just AI that has the capability of destroying the world), you could argue "It needs these 10 capabilities, and I predict it will get each capabilities at these different time periods". If someone actually did that and was historically correct, even within 5-10 years, I'd be impressed and trust their opinion. Have you successfully predicted AI capabilities of the past 10 years?
- possibly mention most recent advances (Transformers, some RL, Neural Nets in 2010's)
2. If there's a 10% chance that it happens during you or your kids lifetime, then that's extremely important. For example, if I thought there was a 10% chance that I died taking the train home tonight, I wouldn't take the train. Even with a 0.1%, I wouldn't.

Though, generally having a set of people working on apologetics for both US & Chinese AI Researchers is potentially extremely high-impact. If you're interested and might be able to donate , DM me for a call.

Next Steps to seriously consider this proposal

Specifying "Solving Alignment"

Part of the problem of alignment is we don't know the correct framework to specify it. The quoted text suggests the criteria for a solution as "a solution alignment researchers can't shoot down.", which side-steps this issue; however, specifying the problem in as fine-grained detail would be extremely useful for communicating to these researchers.

One failure mode would be them taking the money, not get work done, and then argue the problem wasn't specified enough to make any meaningful progress which limits how persuasive this stunt could be. Documents like ELK are more useful specifications that captures the problem to various degrees, and I wish we had more problems like that.

Listing 1000 AI researcher intellectuals

The initial idea is to hire 1000 best AI researchers to work on the problem, not because we expect them to solve it, but by all of them failing

here are a few different proxy's we can use, such as citations and top researchers at AI Labs. So far I've got

1. Authors with the most citations in ML from google scholar
2. DeepMind, Facebook, Microsoft, Tesla, OpenAI, major university groups(?), ?
~~3. Myself and all my friends~~
3.Every author on this Chinese paper

Convincing the CCP and backed researchers is a blank spot in my map, and if anyone knows anything, please comment or message me for a video call.

Actually Hiring People/Creating a Company to Do This

We would need competent people to work on the specification of the problem, outreach and selecting who to pay, keeping up with researchers (if you're paying them $1 million dollars, you also can have 1-on-1 calls which would be useful to make the most of), and reviewing actual work produced(which could actually be done by the community/ independent researchers/orgs).

Timelines Argument

It was argued that these plans are only relevant in 15+ timelines, but huge social changes/cultural norms have happened within 1 year time periods. I'm not giving examples, but they went from, say 50 to 100 to 70 within a few months, which may be significantly different than Alignment.

This Plan May Backfire and Increase Capabilities

A way this can backfire is increasing race conditions, such that everyone wants to create the AGI first. Or at least, more people than are already doing so right now.

I think this is a relevant possibility, and this should be taken into account with whoever reaches out to talk to these top researchers when offering to pay them the large sums of money.

Frontpage

120

Mentioned in

147Reshaping the AI Industry

16Pop Culture Alignment Research and Taxes

Convincing All Capability Researchers

2the gears to ascension

2the gears to ascension

4Alexander Gietelink Oldenziel

New Comment

70 comments, sorted by

top scoring

Click to highlight new comments since: Today at 5:44 PM

[-]Raemon3y780

I do like the core concept here, but I think for it to work you need to have a pretty well specified problem that people can't weasel out of. (I expect the default result of this to be "1000 researchers all come up with reasons they think they've solved alignment, without really understanding what was supposed to be hard in the first place.")

You touch upon this in your post but I think it's kinda the main blocker.

I do think might be a surmountable obstacle though.

6Steven Byrnes3y

I agree. I imagine a disagreement where the person says "you can't prove my proposal won't work" and the alignment judge says "you can't prove your proposal will work", and then there's some nasty, very-hard-to-resolve debate about whether AGI is dangerous by default or not, involving demands for more detail, and then the person says "I can't provide more detail because we don't have AGI yet", etc. I think of the disagreement between Paul and Eliezer about whether IDA was a promising path to safe AGI (back when Paul was more optimistic about it); both parties there were exceptionally smart and knowledgeable about alignment, and they still couldn't reconcile. Giving people a more narrow problem (e.g. ELK) would help in some ways, but there could still be disagreement over whether solving that particular problem in advance (or at all) is in fact necessary to avert AGI catastrophe.

5Raemon3y

I've seen proposals of the form "Eliezer and Paul both have to agree the researchers have solved alignment to get the grand prize", which seems better than not-that, but, still seems insufficiently legibly fair to make it work with 1000 researchers.

2Raemon3y

I have also seem some partially formalized stabs at something like "interpretability challenges", somewhat inspired by Auditing Games, where there are multiple challenges (i.e. bronze, silver and gold awards), and the bronze challenge is meant to be something achievable by interpretability researchers within a couple years, and the gold challenge is meant to be something like "you can actually reliably detect deceptive adversaries, and other key properties a competent civilization would have before running dangerously powerful AGI." This isn't the same as an "alignment prize", but might be easier to specify.

4Linda Linsefors3y

I don't think well specified problem is the way to go. Instead chose a group of judges, and if the majority of them think something is progress, then that counts a winn. This are the rules, and no argument afterwards. But to make this fair, the judges should be available to discuss possible research directions, and what they would and would not concider progress, before and during the 3 months of reaserch. Or if that's not possible, due to the scale, maybe some group of people who are good at predicting what the real judges would say. Besides, giving out the money, and not stay in touch during the 3 months, seems like a bad idea anyway.

2Raemon3y

I agree that actual judges are better for actually solving the problem, the question is whether you can legibly scale the reputation of those judges to 1000 researchers who aren't already bought into the judge's paradigm.

1Linda Linsefors3y

I don't know. I would start small, and worry about scaling later. If this is worth doing at all it is also worth doing at smaller scale, so it would not be a waste if the project don't scale. Also, I suspect that quality is more important than quantity here. Doing a really good job running this program with the 50 most influential AI reserchers, might be better payoff, than doing a half job with the top 1000 reaerchers.

4Raemon3y

Oh, maybe, but it seemed like the specific proposal here was the scale. It seemed like hiring a few people to do AI research was... sort of just the same as all the hiring and independent research that's already been happening. The novel suggestion here is hiring less aligned people at a scale that would demonstrate to the rest of the world the problem was really hard. (not sure exactly what the OP meant, and not sure what specific new element you were was interested in)

4philh3y

I guess a question here is what the returns-to-scale curve looks like? I'd be surprised if the 501-1000th researchers were more valuable than the 1-500th, suggesting there is a smaller version that's still worth doing. I don't know where this guess comes from, but my guess might be that the curve is increasing up to somewhere between 10 and 100 researchers, and decreasing after that. But also there's likely to be threshold effects at round/newsworthy numbers?

2Raemon3y

That does sound right-ish.

1Linda Linsefors3y

I'm not going to speculate about who ment what. But for me the new and interesing idea whas to pay people to do reserch in order to change their mind, as oposed to pay people to do reaserch in order to produce reserch. As far as I know all the current hireing and paying independent reserchers is directed towards people who already do belive that AI Safety reserch is dificult and important. Paying people who are not yet convinced is a new move (as far as I know) even at small scale. I guess that is is currently possible for an AI risk sceptics to get an AI Safety reserch grant. But none of the grant are designed for this purpouse, right? I think the format of a very high payed (more thatn they would earn otherwese) very short (not a major interuption to ongoing research) offer, with the possibility of a price at the end, is more optimised to get skeptics onboard. In short, the design of a funding program will be very diffrent when you have a difrent goal in mind.

[-]Ben Pace3y350

The first issue in my mind is that it’s straightforwardly messing up all business plans those companies and labs have for their staff to leave for 3 months. Leadership will by default be angry and work hard to discredit you and perhaps threaten to fire anyone who accepts your deal.

[-]Raemon3y250

My yes-and thought here is "okay, can we do this in a way that the various labs will feel is exciting, or win-win?". Rather than making the deal with individuals, somehow frame it as a partnership with the companies.

[-]lc3y110

Leadership will by default be angry and work hard to discredit you and perhaps threaten to fire anyone who accepts your deal.

"Why is management so insistent I don't go do this alignment thing if it won't actually change my view?"

This sounds like an wonderful way to invoke the Streisand effect and cause an internal political upheaval. I would love to see DeepMind leadership overreact to this kind of outreach and attempt to suppress a bunch of Google AI researcher's moral compunctions through heavy-handed threats. Just imagine someone doing this if Google researchers had concerns about racial bias in their algorithms. It really seems like the dream scenario.

[-]Ben Pace3y140

I want to explicitly mark this (in my mind) as "babble" of the sort that is "Hufflepuff Bones" from HPMOR. I don't want to have this relationship with the leadership at AI labs. I'm way more interested in finding positive sum actions to take (e.g. trades that AI labs are actively happy about) than adversarial moves.

0lc3y

I want them to stop lighting enormous fires next to open fuel repositories, which is unfortunately what these "leaders" are currently paid millions of dollars to do. Given that, I'm not sure what option there is. They are already adversaries. Nonprofit leadership probably doesn't have much of a vested interest and we should definitely talk to them, but these guys? Seriously, what strategy remains?

[-]Ben Pace3y170

Man, wtf is this argument? Yes you should talk to leaders in industry. I have not yet tried at all hard enough on the "work together" strategy to set it on fire at this moment. I don't think such people they see themselves as adversaries, I don't think we have been acting like adversaries, and I think this has allowed us to have a fair amount of conversation and cooperation with them.

I feel a bit like you're saying "a new country is stockpiling nukes, so let's make sure to quickly stop talking to them". We're in a free market economy, everyone is incentivized to build these companies, not just the people leading existing ones. That's like most of the problem.

I think it's good and healthy to think about your BATNA and figure out your negotiating position, so thinking about this seems good to me, but just because you might be able to exercise unilateral control doesn't mean you should, it lowers trust and the ability for everyone to work together on anything.

I'm not confident here and maybe I should already have given up after OpenAI was founded but I'm not ready to call everyone adversaries, it's pretty damn hard to backtrack on that, and it makes conversation and coordination way way harder.

8lc3y

I think my problem is that I sometimes use "moral culpability" as some sort of proxy for "potential for positive outcomes following dialogue". Should reiterate that it was always my opinion that we should be doing more outreach to industry leaders, even if my hopes are low, especially if it turns out we haven't really tried it. Edit: After further thought I also think the frustration I have with this attitude is: 1. We're not going to convince everybody. 2. Wild success means diverting significant but not necessarily critical amounts of resources (human, monetary, etc.) going toward AI capabilities research toward other less dangerous things. 3. Less AI capabilities research dries up the very short term money. Someone from #1 who we can't convince, or just doesn't care, is going to be mad about this. So it's my intuition that, if you're not willing to annoy e.g. DeepMind's executive leadership, you are basically unable to commit to any strategy with a chance of working. It sucks too because this is the type of project where one bad organization will still end up killing everybody else, eventually. But this is the problem that must be solved, and it involves being willing to piss some people off.

4Ben Pace3y

I am not sure what the right stance is here, and your points seem reasonable. (I am willing to piss people off.)

3Kaj_Sotala3y

There are already perfectly ordinary business reasons for management to not want to lose all their major staff for three months (especially if some of them are needed for fulfilling contractual obligations with existing customers), and the employees have signed job contracts that probably do not provide a clause to just take three months off without permission. So the social expectation is more on the side of "this would be a major concession from management that they're under no obligation to grant" than "management would be unreasonable not to grant this".

4Eli Tyre3y

Thinking out loud about how to address this issue: It doesn't have to be the same 3 months for everyone. You could stagger this over the course of 2 years, with an 8th of the relevant researchers taking a sabbatical to give this challenge a shot, at any given time. This also means that they can build on each other's work, in serial, if they're in fact making progress, and can see how others have failed to make progress, if they're not.

2niplav3y

Maybe alternatively a big EA actor could just buy some big AGI labs? That wouldn't mess up schedules that badly.

2the gears to ascension3y

google's entire stock value is agi

[-]lc3y340

This is a creative idea, and maybe it'll even work, but I do not want us to completely dry up the resources of good-natured people like Sam Bankman-Fried. We could, at the absolute minimum, start this mission with 500k per researcher, one to five well respected researchers internal to these organizations, and a million dollar bonus. A billion dollars is a shit ton of money to waste if something like this doesn't work and I honestly don't think it's wise to set a standard of "asking for it" unless we're literally repelling an asteroid. Six million dollars is a shit ton of money to waste if it doesn't work.

Things aren't that desperate (yet). We have at least a few years to scale up.

4Chris_Leong3y

I expect this proposal would be more likely to receive funding, though normally in funding applications you submit different proposals with various different levels.

[-]Rohin Shah3y290

The capabilities people have thrown a challenge right back at you!

Convincing All Alignment Researchers

Give the world's hundred most respected AI alignment researchers $1M each to spend 3 months explaining why AI will be misaligned, with an extra $100M if by the end they can propose an argument capability researchers can't shoot down. They probably won't make any progress, but from then on when others ask them whether they think alignment is a real unsolved problem, they will be way more likely to say no. That only costs you a hundred million dollars!

I don't think I could complete this challenge, yet I also predict that I would not then say that alignment is not a real unsolved challenge. I mostly expect the same problem from the OP proposal, for pretty similar reasons.

[-]Not Relevant3y110

For the record I think this would also be valuable! If as an alignment researcher your arguments don't survive the scrutiny of skeptics, you should probably update away from them. I think maybe what you're highlighting here is the operationalization of "shoot down", which I wholeheartedly agree is the actual problem.

Re: the quantities of funding, I know you're being facetious, but just to point it out, the economic value of of "capabilities researchers being accidentally too optimistic about alignment" and "alignment researchers being too pessimistic about alignment" are asymmetric.

7Rohin Shah3y

If that's your actual belief you should probably update away now. People have tried to do this for years, and in fact in most cases the skeptics were not convinced. Personally, I'm much more willing to say that they're wrong and so I don't update very much. Yeah, that was indeed just humor, and I agree with the point.

3Not Relevant3y

I was referring to your original statements: I think you might be construing my statement as that you should take AI risk less seriously if you can't convince the skeptics, as opposed to if the skeptics can't convince you.

2Rohin Shah3y

Ah, I see, that makes more sense, sorry for the misunderstanding. (Fwiw I and others have in fact talked with skeptics about alignment.)

6Vaniver3y

Hmmm, I wonder if there's a version like this that would actually be decent at 'getting people to grapple with' the ideas? Like, if you got Alignment Skeptic Alice to judge the ELK prize submissions with Paul Christiano and Mark Xu (presumably by paying her a bunch of money), would Alice come away from the experience thinking "actually ELK is pretty challenging" or would she come away from it thinking "Christiano and Xu are weirdly worried about edge cases that would never happen in reality"?

2Rohin Shah3y

If Alice is a skeptic because she actually has thought about it for a while and come to an opinion, it will totally be the latter. If Alice has just not thought about it much, and she believes that AGI is coming, then she might believe the former. But in that case I think you could just have Alice talk to e.g. me for, say, 10 hours, and I'd have a pretty decent chance of convincing her that we should have more investment in AGI alignment. (Or to put it another way: just actually honestly debating with the other person seems a lot more likely to work than immersing them in difficult alignment research -- at least as long as you know how to debate with non-rationalists.)

1Not Relevant3y

Your time isn’t scalable, though - there are well over 10,000 replacement-level AI researchers.

2Rohin Shah3y

Of course, but "judge the ELK prize submissions with Paul Christiano and Mark Xu" is also not a scalable approach, since it requires a lot of conversation with Paul and Mark?

1Not Relevant3y

I think we agree that scaling is relative and more is better!

4johnlawrenceaspden3y

God, absolutely, yes, do I get to talk to the sceptic in question regularly over the three months? Given three months of dialogue with someone who thinks like me about computers and maths, and where we both promise to take each other's ideas seriously, if I haven't changed his mind far enough to convince him that there are serious reasons to be scared, he will have changed mine. I have actually managed this with a couple of sceptic friends, although the three months of dialogue has been spread out over the last decade! And I don't know what I'm talking about. Are you seriously saying that our best people can't do this?! Eliezer used to make a sport of getting people to let him out of his box. And has always been really really good at explaining complicated thoughts persuasively. Maybe our arguments aren't worth listening to. Maybe we're just wrong. Give me this challenge!! Nobody needs to pay me, I will try to do this for fun and curiosity with anyone on the other side who is open-minded enough to commit to regular chats. An hour every evening? In person would be better, so Cambridge or maybe London? I can face the afternoon train for this.

8Rohin Shah3y

I think it's totally doable (and I have done it myself) to convince people who haven't yet staked a claim as an Alignment Skeptic. There are specific people such as Yann Lecun who are publicly skeptical of alignment research; it is them who I imagine we could not convince.

3johnlawrenceaspden3y

OK, I myself am sceptical of alignment research, but not at all sceptical of the necessity for it. Do you think someone like Eliezer has had a proper go at convincing him that there's a problem? Or will he just not give us the time of day? Has he written anything coherent on the internet that I could read in order to see what his objections are? Personally I would love to lose my doom-related beliefs, so I'd like to try to understand his position as well as I can for two reasons.

8Rohin Shah3y

Here's an example

2johnlawrenceaspden3y

Great, thanks, so I'm going to write down my response to his thoughts as I hear them: Before reading the debate I read the Scientific American article it's about. On first read, that seems convincing, ok, relax! And then take a closer look. What's he saying (paraphrasing stuff from Scientific American): Superintelligence is possible, can be made to act in the world, and is likely coming soon. Intelligence and goals are decoupled Why would a sentient AI want to take over the world? It wouldn’t. Intelligence per se does not generate the drive for domination Not all animals care about dominance. I'm a bit worried by mention of the first law of robotics. I thought the point of all those stories was all the ways such laws might lead to weird outcomes. Blah blah joblessness, military robots, blah inequality, all true, I might even care if I thought there was going to be anyone around to worry about it. But it does mean that he's not a starry-eyed optimist who thinks nothing can go wrong. That's great! I agree with all of that, it's often very hard to get people that far. I think he's on board with most of our argument. And then right at the end (direct quote): OK, so he thinks that because you made a robot, it will stay loyal to you and follow your commands. No justification given. It's not really fair to close read an article in a popular magazine, but at this point I think he maybe he realises that you can make a superintelligent wish-granting machine, but hasn't thought about what happens if you make the wrong wish and want to change it later. (I'm supposed to not be throwing mythological and literary references into things any more, but I can't help but think about the Sybil, rotting in her bag because Apollo had granted her eternal life but not eternal youth, or TS Eliot's: "That is not it at all, That is not what I meant, at all." ) So let's go and look at the debate itself, rather than the article.

2Logan Riggs3y

I was talking to someone recently who talked to Yann and got him to agree with very alignment-y things, but then a couple days later, Yann was saying very capabilities things instead. The "someone"'s theory was that Yann's incentives and environment is all towards capabilities research.

[-]lorepieri3y270

Unpopular opinion (on this site I guess): AI alignment is not a well defined problem, there is no clear cut resolution to it. It will be an incremental process, similar to cybersecurity research.

About the money, I would do the opposite: select researchers that would do it for free, just pay them living expenses and give them arbitrary resources.

7johnlawrenceaspden3y

I don't think that's an unpopular opinion! That's just true. Money-wise, we have a Cassandra problem. No one believes us. So the question is, how can we get everyone competent enough to end the world to realize that they're about to end the world? Is it possible to use money to do this? Apparently there is money, but not so much time.

2the gears to ascension3y

just like parenting

[-]johnlawrenceaspden3y240

Why not do something like the Clay Prize for Mathematics? Offer gargantuan sums for progress on particular subproblems like corrigibility, or inner alignment, or coming up with new reasons why it's even harder than it already looks, or new frameworks for attacking the problem.

Let MIRI be the judge, and make it so that even the most trivial progress gets a fat cheque. (A few thousand for pointing out typos in MIRI papers seems fair)

It should be obviously easy money even to clever undergraduate compscis.

That should inspire vast numbers of people to do the research instead of their day jobs, which would obviously be great news.

That will likely lead to companies banning their staff from thinking about the problem, which is the best possible publicity. Nothing gets one of us thinking like being told to not think about something.

It might even produce some progress (even better!!).

But what it will likely really do is convince everyone that alignment is the same sort of problem as P=NP, or the Riemann Hypothesis.

[-]Joe Collman3y110

This is an interesting idea (of course the numbers might need to be tweaked).

My first concern would be disagreement about what constitutes a solution: what happens if one/many researchers incorrectly think they've solved the problem, and they continue to believe this after alignment researchers "shoot them down", but the solution appears to be correct to most researchers?
Does this backfire and get many to dismiss the problem as already solved?

I think we'd want to set things up with adjudicators that are widely respected and have the 'right' credentials - so e.g. Stuart Russell is an obvious choice; maybe Wei Dai? I'm not the best judge of this.

Clearly we'd want Eliezer, Paul... advising, but for final decisions to be made by people who are much harder to dismiss.

I think we'd also want to set up systems for partial credit. I.e. rewards for anyone who solves a significant part of the problem, or who lays out a new solution outline which seems promising (even to the most sceptical experts).
I'd want to avoid the failure mode where people say at the end (or decide before starting) that while they couldn't solve the problem in 3 months, it's perfectly doable in e.g. a couple of years - o... (read more)

[-]DanielFilan3y100

Give the world's thousand most respected AI researchers $1M each to spend 3 months working on AI alignment, with an extra $100M if by the end they can propose a solution alignment researchers can't shoot down.

An unfortunate fact is that the existing AI alignment community is bad at coming to consensus with alignment solution proposers about whether various proposals have been "shot down". For instance, consider the proposed solution to the "off-switch problem" where an AI maintains uncertainty about human preferences, acts to behave well according to those (unknown) preferences, and updates its belief about human preferences based on human behaviour (such as inferring "I should shut down" from the human attempting to shut the AI down), as described in this paper. My sense is that Eliezer Yudkowsky and others think they have shot this proposal down (see the Arbital page on the problem of fully updated deference), but that Stuart Russell, a co-author of the original paper, does not (based on personal communication with him). This suggests that we need significant advances in order to convince people who are not AI safety researchers that their solutions don't work.

1Not Relevant3y

Strongly encourage you to make this comment its own post; the work of laying out “here’s SOTA and here’s why it doesn’t work” in precise and accessible language is invaluable to incentivizing outsiders to work on alignment.

[-]lc3y100

It was argued that these plans are only relevant in 15+ timelines

Is it OK if I get kinda angry when I think about these constraints, as someone born around 2000? We had 15 years 15 years ago. What was everybody doing when I was 7 years old?

[-]johnlawrenceaspden3y180

They were wanking on endlessly about quantum consciousness and how any agent sufficiently intelligent to escape its box would be intelligent enough to figure out that the utility function we'd given it wasn't actually what we wanted.

I've been going on about 'AI will kill us all' to anyone who'll listen for about ten years now. Everyone I know thinks I've got another stupid science-fiction bee in my bonnet to replace my previous idiotic nerd-concerns about overpopulation and dysgenics and grey goo and engineered plagues and factory farming and nuclear accidents and nuclear proliferation and free speech and ubiquitous surveillance and whatever the hell else it was that used to seem important.

Almost all my friends are graduates of the University of Cambridge, and roughly half of them studied real subjects and can program computers.

Goddamn it, I know actual AI safety researchers, people much cleverer and more capable than me, who just looked at AlphaZero and shrugged, apparently completely unable to see that building a really good general reinforcement learner might have real-world implications beyond chess.

There's some bunch of academic parasites in Cambridge who were originally... (read more)

[-]Chris_Leong3y100

So what's your plan now? Are you going to go down fighting?

1johnlawrenceaspden3y

A good question! I think the answer's no. If there was a bunch of guys around here that I could talk to about this stuff, then I might be able to act as a sort of useful idiot for them to do rubber-duck debugging on. I do still seem to be pretty good at understanding other people's ideas. But there isn't. I have looked. And I never even got my PhD, I don't seem to be very good at generating ideas alone. I'm too old for sport or drinking or womanizing these days, even tennis seems to be too much for my poor knees. I'm even getting bored of computers, to my amazement. But the love of patterns is still there and now there's nothing better to do it seems more and more salient. So I think I'm going to finish reading my current book on undergraduate group theory, and then maybe after that there's something called Visual Differential Geometry that looks cool. And maybe take my narrowboat out into the Summer fens a few last times. Actually MIRI's paper on Garrabrant Inductors looks fascinating, and maybe the last great contribution to the philosophy of mathematics, but I think maybe I need to do an undergraduate logic course first in order to actually understand it. A bit of consulting to pay the rent maybe if we last that long. But it's all zoom calls and open-plan offices these days. Sweaty nerd-pits where the blinds are always down so that the fierce rays of the hated sun don't shine on people's monitors. Can't face that sort of thing in the Summer. Life's to be enjoyed. I never did have the willpower to make myself do something that was no fun.

8Chris_Leong3y

Well, if you change your mind, it'd be great to have you. I'm sure there are some up and coming researchers who'd benefit from having someone else to work through Garrabrant Inductors with them. At the same time, I don't grudge you your decision. I'm glad at least that you seem to have found peace.

5johnlawrenceaspden3y

Chris, if you're speaking on behalf of MIRI, then, to, for once, speak frankly and without irony: If you can pay me some sort of reasonable amount of money for my time (and I mean 'reasonable by academic standards', not 'reasonable by computer programmer standards' and even less 'reasonable by consultant accustomed to astronomical daily rates' standards) to sit around doing roughly what I was going to do anyway, and I don't have to leave my paradise or spend my days in front of a computer, or do anything I don't fancy doing, but I get to sit around in the sunshine reading and thinking and talking to people all day, then I'm yours and always was. I have enough control of my curiosity that I can point it at a copy of 'Logical Induction' and a book on undergraduate logic, if you think that might be a good use of my time in the next few months. And I'm unusually sociable and extrovert for one of us, and I have no fear of public speaking, and I would be quite happy to occasionally get the train to London or Oxford to talk to people if I could help out in that sort of way. Once upon a time I was a promising young mathematician, and I am still rather a good programmer even by the standards of this town. And what I understand I can teach, so if you have any promising young persons who can make it to Cambridge then I can show them what few things I know. But I wonder if you would really want me to do that. I don't want to suck up resources that could be better used elsewhere. I am an old man now, and even as a youngster I was no use at research, and I can practically guarantee that the result of me staring at Logical Induction for long enough will be that I will understand first-year logic, and likely I will understand something of Logical Induction. And that that will be it. I'm not going to solve alignment. I'd be very surprised if I could make any progress on it at all. But if there is a deal to be done here, I will do it. If you want a reference get in touch with

8Vaniver3y

Are you aware of open postings like this one?

2johnlawrenceaspden3y

That looks perfect, thanks. But it also looks like a job offer. If they want a servant they can pay my full commercial rate. And they don't need to, because they can find a much brighter and more idealistic PhD student who'll do it for them for peanuts. So much so that I'm not going to even fill out the form. I don't have the qualifications they're looking for, and that kind of application process is always a sign that it's not worth bothering. Just wasting the time of everyone involved. But if someone recommends me to them (who? why?), and they get in touch and offer to pay me some reasonable retainer (£2000/month?) just to plough through whatever infrabayesianism is under my own power and try to come up with a sensible explanation and some illuminating examples, like I was reading it for fun, then I would totally be up for that. I may fail, of course. And either side can end the deal at any time without notice. So what would they have to lose? Topology and functional analysis were favourites of mine as an undergraduate (30 years ago!!), I've never done measure theory or convex analysis. Background in reinforcement learning theory? Well I read the first half of Sutton and got it, and did all the exercises, and I've been meaning to get round to the second half. But I think that I can still claim to be able to understand complicated maths, especially if I get to talk to the people who came up with it, and sometimes I can teach it to other people or at least help them to understand it.

2Chris_Leong3y

I don't work for MIRI or LessWrong (sadly)[1]. I just meant the community of people who care about these issues. Maybe it was presumptuous for me to use the word "we", but I think people within the community generally do appreciate any contributions that people happen to make. Random fact: I happen to know Ramana Kumar. He was on Australia's International Informatics Olympiad Team with me. 1. ^ I am doing some movement-building in Australia and NZ, but my efforts to encourage people here is kind of out of this scope.

4johnlawrenceaspden3y

My mistake, I use 'we' to talk about us all the time. And I did check to see whether you were MIRI or not, I couldn't tell. You've at least got me to think about my price for lifting a finger to save the world. I am surprised that the answer is 'minimum wage will do as long as I don't have to do anything I don't enjoy'; that seems odd, but I think that is the answer. It feels right. I think at heart the problem is that I don't believe we've got a ghost of a chance. And for me 'dying with dignity' means enjoying what time remains.

2johnlawrenceaspden3y

Staring death in the face and smiling is a character trait of the English. I take no credit for my character class, since I did not choose it. I learned when I was very young that I was going to die, and I saw death very early, and I grew up imagining the nuclear fire blossom over my city, and I have seen friends and mentors and lovers and family die. We were all, always, going to die. Die by fire, die by AI, die by old age, die by disease, what matter?

2lc3y

You could always help us out with this.

2johnlawrenceaspden3y

I've been thinking for a while now that it might be helpful to get my unformed vague intuition that we're all dead into a more legible and persuasive form: https://www.lesswrong.com/posts/xBrpph9knzWdtMWeQ/the-proof-of-doom Certainly I have had little success convincing people that there is even a problem this last decade. Perhaps that could be some use to you?

[-]nem3y60

Another way this could potentially backfire. $1,000,000 is a lot of money for 3 months. A lump sum like this will cause at least some of the researchers to A) Retire, B) Take a long hiatus/sabbatical, or C) Be less motivated by future financial incentives.

If 5 researchers decide to take a sabbatical, then whatever. If 150 of them do? Maybe that's a bigger deal. You're telling me you wouldn't consider it if 5-10 times your annual salary was dropped in your lap?

[-]johnlawrenceaspden3y40

Oh, that's so unfair! I've been avoiding working in AI for years, on account I worried I might have a good enough idea to bring the apocalypse forward a day. One day out of 8 billion lives makes me Worse Than Hitler(TM) by at least an order of magnitude.

And I've even been avoiding blogging about basic reinforcement learning for fear of inspiring people!

Where's my million?!

Also hurry up, I hardly have any use for money, so it will take ages to spend a million dollars even if I get creative, and it doesn't look like we have very long.

9lc3y

This actually brings up an important consideration. It would be bad to incentivize AGI research by paying people who work inside it huge sums, the same way paying slaveowners to free their slaves is a poor abolitionists' strategy. In reality people aren't Homo Economicus, so I think we could work around it, but we'd need to be careful.

[-]Linch3y110

This actually brings up an important consideration. It would be bad to incentivize AGI research by paying people who work inside it huge sums, the same way paying slaveowners to free their slaves is a poor abolitionists' strategy.

? This was literally what the UK did to free their slaves, and (iiuc) historians considered it more successful than (e.g.) the US strategy, which led to abolition a generation later and also involved a civil war.

9johnlawrenceaspden3y

That was after the abolition of slavery by law, and long after the interdiction of the transatlantic slave trade by the Royal Navy. I wonder if it would have been such a good strategy if the compensated could have re-invested their money.

7Linch3y

This is a really good point! Though I thought the subtext of the original comment was about the incentives rather than the causal benefits/harms after receiving the money.

7johnlawrenceaspden3y

That was actually the subtext of my comment, so thanks for making it explicit. To be tediously clear, I think this is a good idea, but only because I take imminent AI extinction doom super-seriously and think just about anything that could plausibly help might as well be tried. As a general policy, I do feel giving all the money you can lay your hands on to the bad guys in proportion to how bad they've historically been is not "establishing correct incentives going forward". (am I supposed to add "according to my model" here? I'm not terribly hip with the crazy rhythm of the current streets.) Not such an issue if there is no going forward. An interesting test case for Logical Decision Theory perhaps?

1JohnBuridan3y

Well, Russia did pay to free the serfs and it would have worked if they didn't design it so that the former serfs had to pay back the debt themselves. Similarly, Ireland paid the big landholders off in a series of acts from 1870 - 1909 which created a new small landholding class. In fact, this type of thing historically seems to work when it concerns purchasing property rights. The analogy though is imperfect for knowledge workers. They don't own the knowledge in the same way. Perhaps, the best way to do this is as previously mentioned, to get a sizable portion of current accomplished researchers to spend time on the problem by purchasing their time.

[-]Alexander Gietelink Oldenziel3y40

This is thinking in the right direction.

Just announcing prize money could be worth it too.

Moderation Log