One of the things I'm most concerned about is the danger that training AIs using human neural activity will teach it to manipulate humans more effectively. Especially when you transition from using neural states as mere observations for the AI to using them as reward targets.
1Hmm, I think I am pretty strongly in favor of mind-reading technology, but weakly held opinion.
The big upside to mind-reading technology, is that we could use it on prospective people in power. I.e. lock powerful positions behind tests that determine whether the candidate in question has good intentions, intends to do what they said they would do, doesn't have ulterior motives people ought to know et cetera. This would solve one of the biggest problems in civillization.
The dictator thing, I don't care that much about it, because I think I'm much more pessimistic than you are about how feasible it is for the populace to overthrow competent dictatorships empowered with modern technology. Like my model of revolutions against oppressive governments, is that the bottleneck (from the peoples perspective), is communication and coordination. And that is something modern governments can suppress effectively without using mind-reading technology.
I also think mind-reading tech would be much more beneficial for making AI go well than you seem to do. Like, in my mind, the primary benefit is not enabling humans staying in the loop for longer, or merging with AIs, or anything like that. It's that...
21Author of the Conduit blog post here. I won't comment on the substance of this post (for now, anyways), but did want to say two meta-level points:
- As usual, LessWrong provides by far the most thoughtful critique I've seen. The comments are also great. In the spirit of Cunningham's Law, I'm glad my blog post provoked this piece!
- I really appreciate OP for emailing me before publishing this post. I asked for a call and heard his threat models beyond what is written here, which were interesting.
4
4I think that's a fine critique, but I think it's good to give people a bunch of time. I do think it's better to indicate you saw something and appreciated it. I think it puts you more on the hook of engaging with it. Serious engagement can follow later on, and of course often takes a while. Some people can shoot off well-formed responses immediately in comments, but most want to take some time to think about how to respond.
1Hopefully, Naomi has already thought about her position against this rather obvious critique (that I'm glad OP made). It's not a surprising subtle flaw in the plan.
Perhaps she requires some time to transmit those thoughts, but a counter-point to "give time" is that it defuses social pressure in the moment where people are ready to apply it. What fraction of people will remember, two months hence, to realize "hey Naomi still hasn't responded" (assuming she hasn't) and dock points? I'd guess the fraction to be rather low.
4The example of a pill that defeats breathalyzers is in tension with the mind reading point. Why are breathalyzers good but mind reading bad? You don't explain.
3If we developed really good mechanistic interpretability I think there is a decent chance a lot of the theory and techniques would transfer to human brains with only moderate adjustment. It could then help build more reliable mind reading technology that can detect deeper and more abstract thoughts in greater detail. This seems maybe bad.
1
You could just as easily make the argument that mind-reading helps out $GOOD_GUYS by rooting out people who would otherwise help $BAD_GUYS. Democratic societies finding anyone with authoritarian or anarchist tendencies. Detecting corrupt officials with morality tests every morning. Your argument that $BAD_GUYS might get the technology is entirely contingent on who you think $BAD_GUYS is.
I would think a much stronger form is the stance that piercing the veil of privacy one has within his own mind is immoral on the face of it, not that it's immoral when used by the wrong people.
1TL;DR: What do you think would be required for mind reading to be safe to develop?
I'm looking into the above question myself from the technological side of things (e.g. security-first software frameworks) since I think we can make progress towards BCI-based human augmentation (and set ourselves up for when it's safe to go further hardware-wise) without actually speeding up malicious use-cases, and also since afaict mind reading is itself an important keystone for both technical alignment and as a safer path to the non-X-risk benefits AI is supposed to pro...
1Presumably there are some number of people N in the world that are wrongfully accused or imprisoned that would be exonerated by the advent of ubiquitous mindreading. There are also cold cases that would be solved and some number of outlaws M that would be brought to justice and prevented from doing further harm.
IDK how big N or M is or how everything would actually play out given all the risks and trade-offs, but I don't think this post is really grappling with them deeply. There are lots of mundane ways that mindreading could reshape society disproportionately for the benefit of the honest and powerless.
1
“I have sworn upon the altar of god, eternal hostility against every form of tyranny over the mind of man”
–Thomas Jefferson, letter to Benjamin Rush
Context: Conduit is building datasets to enable telepathy, to use their term.
I saw my grandfather lose control over his own fingers: what I would have given to offer him a headband that read his thoughts. Through novel technologies we have liberated almost all Americans from farming, driven the child and infant mortality rate from the pre-industrial half to less than half a percent in the best-performing countries, and rendered famine a political choice: broad-based improvements in efficiency are good and should be pursued for their own sake. Telepathy offers more: we could create trust through verified honesty, helping us ensure prosperity and peace. DARPA is already looking into “preconscious” thoughts for suicide prevention. There’s also a strong argument centered on AI Safety: the models are becoming superhuman, and this is technology to allow us to keep pace, minimize hostile competition, and perhaps survive into the future.
This is what Conduit is promising. Unfortunately, mindreading will have other effects.
Oskar Schindler saved over 1,000 Jewish lives during the Holocaust. He did it by lying, frequently, and repeatedly, combined with spending his entire fortune on construction and in the black market: luxuries for corrupt officials and food for his Jewish workers. All of this was against the will of the Third Reich. They launched investigations, and even caught some of the people he was bribing. If you imagine Nazis with mindreading, so they can tell whenever someone is lying to them, there goes Schindler and his employees. Along with him goes every official who suddenly became incapable of spotting forged documents when grandmothers needed to escape disintegrating Yugoslavia, every Polish and Egyptian soldier unwilling to fire on a protesting population, and everyone else who would take the opportunity, in a brutal regime, to show decency and kindness.
The use of technology developed innocently in America for autocratic purposes is not a science fiction story of the previous century: it is a feature of the 21st. IBM sold “policing analysis software” in China that was used for the ongoing genocide in Xinjiang, to pick one of many examples. IBM cut ties and tried to stop distribution, but unmaking technology without unmaking civilization is not yet solved. Back at home, the FBI is looking into predictive AI as senior White House officials clarify that they are an “army” at war with their political opposition, whom they call domestic extremists.
All technologies can be put to harmful ends: the Italian fascists rode trains, which does not imply that trains are bad. Some technologies are credibly broadly good, like washing machines. They replace unpleasant human labor and empower women. There are other technologies that seem mostly bad. If you publicize a recipe for a pill that will defeat a breathalyzer, you have made a tool that is obviously harmful, and no amount of argument or invocations of the rights and obligations of the user will save you from the scorn of all right-thinking people. Tools enhance certain activities, and can implicitly or explicitly discourage others. They can remind you to call your mother, or spend all day doom-scrolling. As such, technical work has moral character (and I really recommend The Moral Character of Cryptographic Work for a better-written explanation).
The problem with mindreading is that it is an asymmetric technology that disproportionately helps those who rule without consent. The productivity enhancements are equally beneficial to everyone who can get them, but detecting defection is of marginal benefit to a project you are joyously working on with friends. It will help large corporations predict who is contemplating leaving. It is of great help to a criminal scared that his subordinates will turn state’s evidence. But it is dictators who will benefit the most.
Subscribe now
The greatest threat to a dictatorship is an information cascade, because a dictatorship can only handle so many rebels at once. This is true for both masses and elites: it’s why one of the first targets of any coup attempt is communications infrastructure. A revolution is a bet that other people agree with you that the leadership is bad, and that you can all rise up together at once. Mindreading lets the autocrat detect people the moment they begin to develop doubts, and “re-educate” or simply kill them. As a result, mindreading is a matter of urgent necessity, backed by the resources of the entire state, for a dictator. Monitoring and punishing dissent isn’t just a potential use: it is one more valuable than anything your merely impressive white-collar worker is doing.
So what does this look like, in 2030, when Conduit expects “invasive general read”? Moving from thought decoding to text to measuring associations with certain words is an active research task with a verifiable outcome and incredible value placed on success. Generals and corporate founders and political elites will get started with their headband in the morning as part of their workday, just like anyone else doing important work, and they will pass a quick quiz on Xi Jinping Thought, just like they have to do every so often. It’s boring, like HR training. If they have a bit too much sympathy for a foreign nation their counter-intelligence service is notified. If they have a bit too much antipathy for the Dear Leader they get told they need reeducation. If there’s far too much antipathy for the dear leader, or much more fondness for anyone politically important over the Dear Leader, they are fired, or worse. They all know this. They all have strategies to make sure that they are fond enough of the Dear Leader.
One of the lessons we have all learned from the labs is that developing a technology for anyone inherently accelerates it for everyone. It is not possible for one company alone to advance the state of the art and, by keeping control, ensure that it will only be used for good purposes. The training of employees in the process of making something is a large factor: the creation of datasets and a supply chain another. Even if a company swears to only use their technology for the most upright of purposes, the technology itself will leak. The possibility of developing a technology does not imply that its arrival is inevitable at a fixed pace: what projects we choose to work on can accelerate or delay technological innovation. The faster you expect the world to change over the next decade, the more valuable it is to delay innovations that would be harmful today.
The benefits to alignment research are real. I’m not convinced they’re large: the decline of centaur chess promises weighs heavily upon any assertion that human-AI teaming will reliably beat AI models alone. More concerningly, you need a great deal of confidence in your model of the world to be confident that the benefits outweigh the costs of empowering dictatorships. I don’t have that confidence. I’m not convinced anyone should.
And so I ask you to not build the mindreading datasets. Do not build the mindreading hardware. Do not fund this technology. Do not sell them your mind. I am not calling for a boycott after the point where it becomes commercially available: that is far too late to be useful, and I will not begrudge anyone their personal productivity tools. But we should not accelerate mindreading. If it looks at thoughts or feelings, it is a tool for tyrants.
I will not speculate about what dictators might do with mind-writing.
Share