Sorted by New

Wiki Contributions


MIRI announces new "Death With Dignity" strategy

While we happen to be on the topic: can I ask whether (a) you've been keeping up with Vanessa's work on infra-Bayesianism, and if so, whether (b) you understand it well enough to have any thoughts on it? It sounds (and has sounded for quite a while) like Vanessa is proposing this as an alternative theoretical foundation for agency / updating, and also appears to view this as significantly more promising than the stuff MIRI has been doing (as is apparent from e.g. remarks like this):

Optimism about deep learning: There has been considerable progress in theoretical understanding of deep learning. This understanding is far from complete, but also the problem doesn't seem intractable. I think that we will have pretty good theory in a decade, more likely than not[...]

Yudkowsky seems to believe we are pretty far from a good theory of rational agents. On the other hand, I have a model of how this theory will look like, and a concrete pathway towards constructing it.

Ideally I (along with anyone else interested in this field) would be well-placed to evaluate Vanessa's claims directly; in practice it seems that very few people are able to do so, and consequently infra-Bayesianism has received very little discussion on LW/AF (though my subjective impression of the discussion it has received is that those who discuss it seem to be reasonably impressed with / enthusiastic about it).

So, as long as one of the field's founding members happens to be on LW handing out takes... could I ask for a take on infra-Bayesianism?

(You've stated multiple times that you find other people's work unpromising; this by implication suggests also that infra-Bayesianism is one of the things you find unpromising, but only if you've been paying enough attention to it to have an assessment. It seems like infra-Bayesianism has flew under the radar of a lot of people, though, so I'm hesitantly optimistic that it may have underflew your radar as well.)

Ukraine #3: Decision Theory, Madman Theory and the Mafioso Nature

So, I did say in my previous comment that my analyses were merely surface-level, and it was entirely plausible to me that "more sophisticated" analysis would overturn them... but unfortunately, after reading your comment, I have to say: more sophisticated analysis this is not. To begin with:

It seems like Russian and Chinese psyops have really managed to subvert the narrative here in the West. Suddenly, economic sanctions and blockades are seen as "escalation", "aggressive military action" and "insane". But blatantly invading peaceful countries minding their own business and murdering tens of thousands of people? That's just "defending our security interests", and somehow it's all NATO's fault anyway. The funny thing is you would usually expect this sort of double-standard word games from the dominant power, not from the runner-up. In the end, all the world's military might is worth nothing if one has no spine.

Fundamentally, a "naval blockade" is an activity wherein one country's ships engage in physical interference with another country's ships. In general, physical interference is viewed as an escalatory move, and physical interference backed by the threat of physical violence even more so. That a sufficiently tense standoff could become a casus belli follows from this fairly directly (and, in fact, such an outcome is not without historical precedent).

I don't consider the above logic particularly difficult or subtle; nor, it seems to me, is this line of argument anywhere addressed in the above quote. Instead, I see what looks to me like a series of gratuitous tu quoque arguments, many of which impute to me positions which I at no point even gestured toward in my previous comment, much less endorsed.

I will be blunt: this is not virtuous epistemic behavior, especially on a site such as LW. (Yes, what Russia is doing in Ukraine is bad; no, that has nothing to do with whether imposing a US naval blockade on China's maritime trade would be a safe move. I am amazed I have to clarify this.)

India is also the second largest grain consumer in the world, and is effectively neutral with regard to import/export. India's net grain export amounts to ~400,000 tons, while China imports nearly 10 million tons. That is assuming China's domestic production remains constant, which it most definitely will not because agriculture is a heavily industrialized sector. Oil and gas aren't just for keeping the lights on, they're vital in the production and distribution of food.

There look to be a number of strange assumptions in this paragraph. To start with, the latter half appears to take as given the notion that China will begin running out of oil and gas at some point (why? how?), and uses this premise (which is nowhere justified or even further discussed in the entirety of the remaining comment) as a starting point to argue that its domestic grain production would be affected.

The first half, meanwhile, seemingly takes for granted India's 2018-19 export numbers, while ignoring the fact that India has always had a large surplus of grain relative to the amount exported—a surplus which is only growing as time passes. The reason, then, that India's export numbers have not grown alongside its production surplus is that it has managed to capture relatively few foreign markets, with its main customers being far smaller neighboring countries such as Bangladesh. If new markets open up, India has more than the necessary supply to increase its exports drastically; and, in fact, producers are already gearing up to do so.

That land border between China and India by the way? It's the friggin' Himalayas; the most impassable terrain on the planet. Trade flow through there is utterly negligible. Disputes over that very border is also the root of frosty relationship between the two countries since the 1960's, ironically, which brings me to the next point: Why in the world would India come to China's aid? Aside from Pakistan (which is China's closest ally), China is the rival India is facing on the world stage. If anything, India is more likely to be the one doing the blockade.

Starting once more from the bottom rather than the top: India's trade relations with China have always been decent, geopolitical rivalry notwithstanding. The notion that geopolitical concerns would prevent producers from going where the money lies is itself a strange one; not long ago, for example, India explicitly refused to mirror Western sanctions on Russia despite significant pressure to do so, and have since signed several additional trade agreements with Russia—and on the China front, they hosted a visit from China's foreign minister less than a week ago, right after declining a similar visit from a UK delegation. And, considering both countries' participation in BRICS, it seems like utterly wishful thinking to believe that India would refuse access to China's wheat and grain markets should they become available, much less that they would impose a naval blockade against said country.

The point about the Himalayas, on the other hand, is well taken; it's true that trade along those routes is mostly negligible (although I wouldn't count it as anything close to assured that things will remain that way, if circumstances necessitate otherwise). However, the issue is largely moot, as reliance on a land-based trade route would only be required, again, if a naval blockade against trade between India and China were imposed, and that argument remains as ridiculous as it was when it was first put it forth.

Russia doesn't look much better. Sure, there is the Trans-Siberian railway, but it is already running at capacity, and it still is pitiful compared with maritime trade (~1%). There is a good reason why maritime powers have ruled the world for the last 500 years. The oil pipelines going from Siberia to China isn't even connected to the rest of the Russian pipeline network, which why all the recent talks about China supplanting Europe as Russia's main energy customer is nothing but hot air at least in the short term. Thousands of kilometers of pipelines through Siberia isn't built in a day. Looking at the mid term though, it could go either way. Maybe trade ties between China and Russia will deepen, or there could be a regime change pushing Russia towards the West, or Russia may collapse altogether as a nation and descend into internal ethnic conflicts. It's anyone's guess depending how this war with Ukraine goes.

You're absolutely right that critical infrastructure isn't built in a day, but it's that precise fact that allows one to get a read on what countries' intentions are before those intentions become a physical reality. And in point of fact, Russia is breaking more ice in Siberia, a move whose purpose can only point in one direction. It seems to me that, short of a full regime change in Moscow (which itself seem to me like wishful thinking), closer relations between China and Russia are basically an inevitability; and given that that's the case, the Western sanctions against Russian exports can only benefit China, who gains the leverage to purchase those exports at a large discount. I don't see how you argue otherwise—literally, as in: you do not argue otherwise, anywhere in your comment that I can see.

Concluding Thoughts

That's it for the object-level points. On a higher level, though: I want to note that it's not clear to me what you're trying to argue anymore. A lot of the things you wrote read like... well, to be blunt, it reads like like what you're really trying to express is "yay US, boo Russia / China!"—which would explain why, for example, you brought up Russia's atrocities in Ukraine, apropos of nothing, in a discussion about whether physically blockading Chinese trade vessels would be a good idea.

And, like, if that's what you want to express, then... great? I share those sentiments too: obviously it'd be a great outcome if the morally better countries triumphed by default, because the nasty authoritarian countries imploded due to worse management / demographics / whatever. (Or because the US exercised its magical capability to impose maritime trade blockades on near-peer powers without retaliation.) But if you let that sentiment leak into your object-level analysis, I don't think the results will be all that pretty.

It Looks Like You're Trying To Take Over The World

FWIW, I did actually manage to guess which Mark it was based on the content of the initial comment, because there aren't that many persistent commenters named Mark on LW, and only one I could think of who would post that particular initial comment. So claiming not to have deanonymized him at all does seem to be overstating your case a little, especially given some of your previous musings on anonymity. ("The lady doth protest too much, methinks" and all that.)

I do, however, echo the sentiment you expressed on the EA Forum (that anonymous commenting on LW seems not worth it on the margin, both because the benefits themselves seem questionable, and because it sounds like a proper implementation would take a lot of developer effort that could be better used elsewhere).

It Looks Like You're Trying To Take Over The World

(and to you, specifically, Mark*)

* Don't worry. I'm sure LW2 is the only software from here on out which will have silly security bugs.

...Okay, I admit to some curiosity as to how you pulled that one off, though not enough curiosity to go poking around myself in the codebase. Is this one of those things where an explanation (public or private) can be given, at least after the vulnerability is patched (if not before)?

Ukraine #3: Decision Theory, Madman Theory and the Mafioso Nature

A lot of the things you wrote run contrary to the general impression I have of the situation. (Admittedly, I haven't been paying very much attention to the situation, but still...)

Civilian ships are not heavily armed. They can be stopped without violence; it's called a naval blockade.

It seems really obvious to me that this is the kind of measure that would be perceived as a clear escalation? It doesn't really matter whether you're sinking the ships or just blockading them; if you're actively making moves to prevent unaffiliated vessels from other countries from navigating international waters, that seems like a pretty central example of "aggressive military action", and would almost certainly invite retaliation in kind. Given that, is there some interpretation of this quote that isn't as insane as it sounds?

Agricultural products and energy are commodities that China would be more able to acquire from countries less aligned with the USA.

They most definitely can't. China represents one fifth of world population; there is no viable replacement at this scale that doesn't also rely on maritime transport, which is why a huge population is a vulnerability.

That definitely sounds wrong to me. A cursory look reveals that the top three producers of wheat in the world are China, Russia, and India, in that order; collectively these three countries account for ~40% of the world's wheat, and the trade relations between said three countries have been quite clear for a while now, but have especially crystallized over the last few weeks. China in particular shares land borders with both India and Russia, which minimizes the need for "maritime transportation", especially with Russia's recent forays into the Arctic. As for energy, here are the rankings for oil and natural gas, for which the amount of procurement in non-US-aligned countries is (again) substantial, with Russia once more leading the charge in both categories.

At least at first blush, this seems like it cuts pretty hard against your core claim. It's certainly possible that a more sophisticated analysis might reveal complicating factors, but if so I'd ask what specifically you think those complicating factors are, and how they overturn the case outlined above.

Meanwhile, a significant chunk of the USA's living standards and technology infrastructure depends on imported Chinese goods. If the USA decided to sanction China, it would be a more severe blow both to ordinary people's standard of living and a threat to the USA's technological dominance than any impacts we're suffering from sanctioning Russia. 

I perceive these statements as not only false, but backwards. The US initiated a trade war with China as a response to threats to our technological dominance, and together with the pandemic has led to our current period of rapid economic de-globalization, the effect of which has been the fastest re-industrialization in the US since WWII and record-low unemployment numbers since normalized relationship with the PRC. All of which is past and present tense, i.e. not even a prediction. Neither is it partisan talking points; the course has stayed pretty much the same under both Trump and Biden.

I don't think you've done a good job establishing causality here. It's all well and good to post unemployment numbers, but without something tying those numbers to the trade war, it's not clear to me what the right conclusion to draw actually is. This is doubly true considering that the source you posted actually shows approximately constant unemployment from 2017 onward (discounting the obvious blip in 2020), which means the timing doesn't match up to the trade war started by the Trump administration.

I've also seen sources claiming that the trade war lost rather than created jobs on net; here's an example of an article from January 2021 asserting exactly that. Given this, I think it's an open question whether things like trade wars (or other forms of deglobalization) are more likely to hurt or help; and for what it's worth my instinct is definitely on the "hurt" side more than the "help" side. Once more, if you have a sophisticated argument for why this isn't the case, I'd love to hear it.

Ukraine Post #2: Options

I admit to not being super interested in the larger geopolitical context in which this discussion is embedded... but I do want to get into this bit a little more:

think as little as possible before making a commitment in your mind, because if you think more, you might conclude (via simulation or abstract reasoning) that the other player already made their commitment so now your own decision has to condition on that commitment

It's not obvious to me why the bolded assertion follows; isn't the point of "updatelessness" precisely that you ignore / refrain from conditioning your decision on (negative-sum) actions taken by your opponent in a way that would, if your conditioning on those actions was known in advance, predictably incentivize your opponent to take those actions? Isn't that the whole point of having a decision theory that doesn't give in to blackmail?

Like, yes, one way to refuse to condition on that kind of thing is to refuse to even compute it, but it seems very odd to me to assert that this is the best way to do things. At the very least, you can compute everything first, and then decide to retroactively ignore all the stuff you "shouldn't have" computed, right? In terms of behavior this ought not provide any additional incentives to your opponent to take stupid (read: negative-sum) actions, while still providing the rest of the advantages that come with "thinking things through"... right?

and by thinking more you also make it harder for the player to conclude this about yourself

This part is more compelling in my view, but also it kind of seems... outside of decision theory's wheelhouse? Like, yes, once you start introducing computational constraints and other real-world weirdness, things can and do start getting messy... but also, the messiness that results isn't a reason to abandon the underlying decision theory?

For example, I could say "Imagine a crazy person really, really wants to kill you, and the reason they want to do this is that their brain is in some sense bugged; what does your decision theory say you should do in this situation?" And the answer is that your decision theory doesn't say anything (well, anything except "this opponent is behaviorally identical to a DefectBot, so defect against them with all you have"), but that isn't the decision theory's fault, it's just that you gave it an unfair scenario to start with.

What, if anything, am I missing here?

On presenting the case for AI risk

Strong upvote; agree with most / all of what you wrote. Having said that:

Isn't this immediately falsified by human beings? ... And isn't it a bit concerning if your alleged generalization breaks down hardest on the most relevant data point we have for trying to predict the impact of automating general intelligence?

I'm not sure how Conor would reply to this, but my models of Paul Christiano and Robin Hanson have some things to say in response. My Paul model says:

Humans were preceded on the evolutionary tree by a number of ancestors, each of which was only slightly worse along the relevant dimensions. It's true that humans crossed something like a supercriticality threshold, which is why they managed to take over the world while e.g. the Neanderthals did not, but the underlying progress curve humans emerged from was in fact highly continuous with humanity's evolutionary predecessors. Thus, humans do not represent a discontinuity in the relevant sense.

To this my Robin model adds:

In fact, even calling it a "supercriticality threshold" connotes too much; the actual thing that enabled humans to succeed where their ancestors did not, was not their improved (individual) intelligence relative to said ancestors, but their ability to transmit discoveries from one generation to the next. This ability, "cultural evolution", permits faster iteration on successful strategies than does the mutation-and-selection procedure employed by natural selection, and thus explains the success of early humans -- but it does not permit for a new-and-improved AGI to come along and obsolete humans in the blink of an eye.

Of course, I have to give my Eliezer model (who I agree with more than either of the above) a chance to reply:

Paul: It's all well and good to look back in hindsight and note that some seemingly discontinuous outcome emerged from a continuous underlying process, but this does not weaken the point -- if anything, it strengthens it. The fact that a small, continuous change to underlying genetic parameters resulted in a massive increase in fitness shows that the function mapping design space to outcome space is extremely jumpy, which means that roughly continuous progress in design space does not imply a similarly continuous rate of change in real-world impact; and the latter is what matters for AGI.

Robin: From an empirical standpoint, AlphaGo Zero is already quite a strong mark against the "cultural evolution" hypothesis. But from a more theoretical standpoint, note that (according to your own explanation!) the reason "cultural evolution" outcompetes natural selection is because the former iterates more quickly than the latter; this means that it is speed of iteration that is the real underlying driver of progress. Then, if there exists a process that permits yet faster iteration, it stands to reason that that process would outcompete "cultural evolution" in precisely the same way. Thinking about "cultural evolution" gives you no evidence either way as to whether such a faster process exists, which essentially means the "cultural evolution" hypothesis tells you nothing about whether / how quickly AGI can surpass the sum total of humanity's ability / knowledge, after being created.

Ukraine Post #2: Options

This simply kicks the buck to the question of why the US (and other countries) seem reluctant in general to make long-lasting treaties, and the answer to that question still seems pretty clearly some form of "lack of ability (desire) to keep (make) commitments that are (will be) no longer seen as in our interest in the face of pressure".

Also, I would argue that the Trump administration did in fact significantly weaken the US' credibility even on the holding-up-existing-commitments front, what with their treatment of various agreements like NAFTA or the Trans-Pacific Partnership; this shows fairly starkly that even agreements that are supposed to bind future iterations of the same government do not in fact do so, which in turn implies that there is no real mechanism to make those kinds of promises at all, regardless of whether you call them "treaties", "agreements", "pacts", or what have you.

Unlike Zvi, I am less inclined to treat this as anyone's fault in particular; it certainly isn't just unique to the US. But it is an unfortunate consequence of the way humans, even (especially?) humans in power, seem to implement CDT by default (and an extremely myopic version of CDT at that). I don't think it's a stretch to argue smarter agents would do better here.

We're already in AI takeoff

I think the point under contention isn't whether current egregores are (in some sense) "optimizing" for things that would score poorly according to human values (they are), but whether the things they're optimizing for have some (clear, substantive) relation to the things a misaligned AGI will end up optimizing for, such that an intervention on the whole egregores situation would have a substantial probability of impacting the eventual AGI.

To this question I think the answer is a fairly clear "no", though of course this doesn't invalidate the possibility that investigating how to deal with egregores may result in some non-trivial insights for the alignment problem.

It Looks Like You're Trying To Take Over The World

After a quick look at some of this user's other comments and posts, I would like to register, for the purpose of establishing common knowledge, that this is a user whose further contributions to LW I, personally, do not much desire.

Load More