trevor

Not to be confused with the Trevor who works at Open Phil.

Sequences

AI Manipulation Is Already Here

Wiki Contributions

Comments

Ah, neat, thanks! I had never heard of that paper or the Conger-Kanungo scale, when I referred to charisma I intended it in the planecrash sense of charisma that's focused on social dominance and subterfuge, rather than business management which is focused on leadership and maintaining the status quo which means something completely different and which I had never heard of.

The application of variance, to the bundle of traits under the blanket label charisma (similar to the bundle of intelligence and results-acquisition under the blanket label thinkoomph), and the sociological implications of more socially powerful people being simultaneously more rare and also more capable of making the people around them erroneously feel safe, was a really interesting application that I picked up almost entirely from planecrash, yes.

I think that my "coordination takeoffs" post also ended up being a bad example for what you're trying to gesture at here, I already know what I got wrong there and it wasn't that (e.g. basically any China Watcher who reads and understands most of Inadequate Equilibria is on course towards the top of their field). Could you try a different example?

The Cold War analogy is a bit hard to work with, mainly because the original Cold War was a specific state of paradigms that largely can't repeat; we have computers everywhere, thriving international trade and growth, and more importantly, the original Cold War emerged out of the World War paradigm and was started with intent to use nuclear weapons for carpet bombing (this is where the word "WW3" came from), whereas we now have norms and decades of track record of nuclear brinkmanship and de-escalation (the Cold War was established largely due to everyone everywhere having zero experience with this). 

It's similar to expecting the World War paradigm to return, but not nearly as bad, since most people in power in governments and militaries today came of age during the original Cold War and can easily imagine their world becoming more like that again.

Yes, this is why I put "decentralized" in the title even though it doesn't really fit. What I was going for with the post is that you read it yourself, except whenever the author writes about law, you think for yourself about stacking the various applications that you care about (not courts) with the complex caveats that the author was writing about (while they were thinking about courts). Ideally I would have distilled it as the paper is a bit long.

This credibly demonstrates that the world we live in is more flexible than it might appear. And on the macro-civilizational scale, this particular tech looks like it will place honest souls higher-up on net, which everyone prefers. People can establish norms of remaining silent on particular matters, although the process of establishing those norms will be stacked towards people who can honestly say "I think this makes things better for everyone", "I think this is a purity spiral" and away from those who can't.

At work, you could expect to be checked for a "positive, loyal attitude toward the company" on as frequent a basis as was administratively convenient. It would not be enough that you were doing a good job, hadn't done anything actually wrong, and expected to keep it that way. You'd be ranked straight up on your Love for the Company (and probably on your agreement with management, and very possibly on how your political views comported with business interests). The bottom N percent would be "managed out".

This is probably already happening.

There's bad actors who infiltrate, deceptively align, move laterally, and purge talented people (see Geeks, Mops, and Sociopaths) but I think that trust is a bigger issue. 

High-trust environments don't exist today in anything with medium or high stakes, and if they did then "sociopaths" would be able to share their various talents without being incentivized to hurt anyone, geeks could let more people in without worrying about threats, and people could generally evaluate each other and find the place where their strengths resonate with others.

That kind of wholesome existence is something that we've never seen on Earth, and we might be able to reach out and grab it (if we're already in an overhang for decentralized lie detectors).

This is actually a dynamic I've read a lot about. The risk of ending up militarily/technologically behind is already well on the minds of the people who make up all of the major powers today, and all diplomacy and negotiations are already built on top of that ground truth and mitigating the harm/distrust that stems from it. 

Weakness at mitigating distrust = just being bad at diplomacy. Finding galaxy-brained solutions to coordination problems is necessary for being above par in this space.

[Caveat lector: I know roughly nothing about policy!]

For AI people new to international affairs, I've generally recommend skimming these well-respected texts that are pretty well-known to have directly inspired many of the people making foreign policy decisions:

  • Chapters 1 and 2 of Mearsheimer's Tragedy of Great Power Politics (2010). The model (offensive realism) is not enough by itself, but it helps to start with a flawed model because the space is full of them, this model has been predictive, it's popular among policymakers in DC, and gives a great perspective on how impoverished foreign policy culture is because nobody ever reads stuff like the Sequences.
  • Chapters 1 and 4 of Nye's Soft Power (2004)(skim ch. 1 extra fast and ch. 4 slower). Basically a modern history of propaganda and influence operations, except cutting off at 2004. Describes how the world is more complicated than Tragedy of Great Power Politics describes.
  • Chapters 1 and 2 of Schelling's Arms and Influence (1966). Yes, it's that Schelling, this was when he started influencing the world's thinking about how decision theory drives nuclear standoffs, and diplomacy in general, in the wake of the Cuban Missile Crisis. You can be extremely confident that this was a big part of the cultural foundation of foreign policy establishments around the world, plus for a MIRI employee it should be an incredibly light read applying decision theory to international politics and nuclear war. 

I'm going to read some more stuff soon and possibly overhaul these standard recommendations.

Akash also recommended Devil's Chessboard to understand intelligence agencies, and Master of the Senate and Act of Congress to understand Congress. I haven't gotten around to reading them yet, and I can't tell how successful his org has been in Congress itself (which is the main measurement of success tendency), but the Final Takes section of his post on Congress is fantastic and makes me confident enough to try them out.

My thinking about this is that most people usually ask the question "how weird does something have to be until it's not true anymore", or less likely to be true, and don't really realize that particle physics already demonstrated long ago that there just isn't a limit at all.

I was like this for an embarrassingly long time; lightcones and Grabby Aliens, of course that was real, just look at it. But philosophy? Consciousness ethics? Nah, that's a bunch of bunk, or at least someone else's problem.

I went back and tried playing it again, and I'm no longer confident in Universal Paperclips. It's way too heavy on the explore aspect of the explore-exploit tradeoff; you're constantly bombarded with new things to try and have no way of knowing how much they're helping you (maximizing things other than paperclips is usually the winning strategy). It probably doesn't outperform speedrunning most things e.g. various parts of TOTK.

The question-asker here looks too much like a caricature. This might be more representative of people in the real world, but it still gives off a bad vibe here. 

I recommend making the question-asker's personality look more like the question-asker in Scott Alexander's Superintelligence FAQ. Should be a quick fix.

Great image, BTW! I don't think it's the final form but it's a great idea worthy of refinement.

Load More