My Assessment of the Chinese AI Safety Community

[-]lionhearted (Sebastian Marshall)2y*413

I'm a Westerner, but did business in China, have quite a few Chinese friends and acquaintances, and have studied a fair amount of classical and modern Chinese culture, governance, law, etc.

Most of what you're saying makes sense with my experience, and a lot of Western ideas are generally regarded as either "sounds nice but is hypocritical and not what Westerns actually do" (a common viewpoint until ~10 years ago) with a later idea of "actually no, many young Westerners are sincere about their ideas - they're actually just crazy in an ideological way about things that can't and won't work" that is a somewhat newer idea. (白左, etc)

The one place I might disagree with you is that I think mainland Chinese leadership tends to have two qualities that might be favorable towards understanding and mitigating AI risk:

(1) The majority of senior Chinese political leadership are engineers and seem intrinsically more open to having conversations along science and engineering lines than the majority of Western leadership. Pathos-based arguments, especially emerging from Western intellectuals, do not get much uptake in China and aren't persuasive. But concerns around safety, second-order effects, th... (read more)

[-]Lao Mein2y104

I fully endorse this post.

[-]Haoxing Du2y357

Thanks for writing this post! I want to note a different perspective. Although unlike OP, I have not lived in China since 2015 and am certainly more out of touch with how the country is today.

I do observe some of the same dynamics that OP describes, but I want to point out that China is a really big country with inherently diverse perspectives, even in the current political environment. I don’t see the dynamics described in this post as necessarily the dominant one, and certainly not the only one. I know a lot of young people, both in my social circle and online, that share many of the Western progressive values such as the pursuit of equality, freedom, and altruism. I see many people trying their best to live a meaningful life and do good for the world. (Of course, many people are not thinking about this at all, but that is the same everywhere. It's not like these concerns are that mainstream in the West.) As a small piece of evidence, 三联生活周刊 did an interview with me about AI safety recently, and it got 100k+ views on WeChat and only positive comments. I’ve also had a few people reach out to me expressing interest in EA/AI safety since the interview came out.

You can't just hope

... (read more)

[-]simeon_c2y2721

Thanks for writing this.

Overall, I don't like the post much under it's current form. There's ~0 evidence (e.g. from Chinese newspapers) and there is very little actual argumentation. I like that you give us a local view but putting a few links to back your claims would be very very appreciated. Right now it's hard to update on your post given that the claims are very empirical and without any external sources.

More minorly: "A domestic regulation framework for nuclear power is not a strong signal for a willingness to engage in nuclear arms reduction" I also disagree with this statement. I think it's definitely a signal.

[-]Cervera2y252

A "moonshot idea" I saw brought up is getting Yudkowsky's Harry Potter fanfiction translated into Chinese (please never ever do this).

Can you expand on this? Why would it be a bad idea? I have interacted with mainland chinese people (outside of china) and I'm not really making the connection.

[-]Lao Mein2y10644

Let's just say that weirdness in China is very different from weirdness in the West. AI safety isn't even a weird concept here. It's something people talk about, briefly think over, then mostly forget, like Peter Thiel's new book. People are generally receptive to it. What AI safety needs to get traction in the Chinese idea sphere is to rapidly disassociate with really really weird ideas like EA. EA is like trying to shove a square peg into the round hole of Chinese psychology. It's a really bad sign that the AI Safety toehold in China is clustered around EA.

Rationality is pretty weird too, and is honestly just extra baggage. Why add it to the conversation?

We don't need rationality or EA to get Chinese to care about AI safety. Trying to import the Western EA-AI safety-Rationality memeplex wholesale is both unnecessary and detrimental.

[-]avturchin2y426

Interestingly, Yud is attractive to Russian mindset (similarly to Karl Marx). I heard 12 old children discussing HPMOR on the beach, and their parents were not rationalists.

[-]RomanS2y149

From my observations, the Chinese mindset is much more different from the American one than the Russian mindset.

In comparison to Chinese, Russians are just slightly unusual Europeans, with mostly the same values, norms, worldviews as Americans.

I attribute it to the 3000+ years of a relatively strong cultural isolation of China, and to the many centuries of Chinese rulers running all kinds of social engineering (including the purposeful modification of the language according to political goals).

[-]ryan_b2y185

We don't need rationality or EA to get Chinese to care about AI safety. Trying to import the Western EA-AI safety-Rationality memeplex wholesale is both unnecessary and detrimental.

I agree with this, not out of any particular expertise on China, but instead because porting a whole memeplex across a big cultural boundary is always unnecessary and detrimental.

[-]trevor2y1110

I just want to note that rationality can fit into the Chinese idea sphere, very neatly; it's just that it's not effortless to figure out how to make it work.

The current form e.g. the sequences, is wildly inappropriate. Even worse, a large proportion of the core ideas would have to be cut out. But if you focus on things like human intelligence amplification and forecasting and cognitive biases, it will probably fit into the scene very cleanly.

I'm not willing to give any details, until I can talk with some people and get solid estimates on the odds of bad outcomes, like the risk that rationality will spread, but AI safety doesn't, and then the opportunity is lost. The "baggage" thing you mentioned is worth serious consideration, of course. But I want to clarify that yes, EA won't fit, but rationality can (if done right, which is not easy but also not hard), please don't rule it out prematurely.

2RomanS2y

I agree. Some starting points (kudos to GPT-4): Confucius (551-479 BCE) - taught that people should think critically and rationally about their actions and decisions in order to lead a life of harmony and virtue. One of his famous sayings is, "When you know a thing, to hold that you know it; and when you do not know a thing, to allow that you do not know it - this is knowledge." Mencius (372-289 BCE) - believed that individuals can cultivate their moral and intellectual capabilities through rational thinking and learning. Mencius emphasized the importance of moral reasoning and introspection in making ethical decisions. Mozi (470-391 BCE) - advocated for a rational and pragmatic approach to decision-making. He argued that people should evaluate the potential consequences of their actions based on the benefits or harms they would bring to society. Mozi's philosophy encouraged rational thinking and objective analysis in the pursuit of social harmony and the greater good. Zhuangzi (369-286 BCE) - believed that individuals should cultivate their understanding of the natural world and develop their innate abilities to think and reason. Zhuangzi encouraged the cultivation of a clear and unbiased mind in order to achieve harmony with the Dao, or the natural order of the universe. Xunzi (312-230 BCE) - believed that people must be taught to act morally and rationally. Xunzi emphasized the importance of education, self-discipline, and reflection in developing moral character and rational decision-making abilities.

7Meena Kumar2y

I think a similar thing is true in India.

4Sean H2y

I think the most surprising part about your post (and the best part I guess) is discovering how many people in the West have a very poor understanding of how the CCP (and Chinese politics) work. Do you have good newsletters / sites that people could follow?

5lb_rv2y

Not about the CCP or politics but I've found Chinese Doom Scroll tremedously useful as a window into Chinese culture and ways of thinking. It's a daily translation of popular Weibo posts that the author encouters while doom scrolling.

4Algon2y

I wonder if Yud would be willing to write a rat fic aimed at Chinese audiences? He seems to read a bunch of Xinxia, so he's probably absorbed some of the memes of China's male youth. Maybe a fanfic of "Oh my god! Earthlings are insane!" would be a good choice, based on my impression of the novel's themes and what its readership is like. EDIT: I think the rationality angle is important for making progress on AI safety, but I'm not sure which parts are necessary. Also, what part of HPMOR would make it especially bade for Chinese audiences? The libertarian sympathies? The trans-humanism doesn't seem like it would be that harmful, given the popularity of novels Embers ad infinatum. Which is another novel that Yud could write a fanfic for.

[-]Lao Mein2y3821

The most common response I get when I talked to coworkers about AI risk wasn't denial or an attempt to minimize the problem. It was generally something like "That sounds really interesting. If a company working on the problem was paying a lot, I would consider jumping ship." And then a shrug before they went back to their tasks. I don't see how rationality helps with anything. We know what the problem is, and just want to be paid to solve it.

I can't really explain why HPMOR is insanely cringe in a Chinese context to someone without the cultural background. It's not something you can argue people out of. Just trust me on this one.

[-]Tomás B.2y2611

Is it "insanely cringe" for different reasons than it is "insanely cringe" for English audiences? I suspect most Americans, if exposed to it, would describe it as cringe. There is much about it that is cringe, and I say this with some love.

[-]rank-biserial2y207

"That sounds really interesting. If a company working on the problem was paying a lot, I would consider jumping ship."

The Chinese stated preferences here closely track Western revealed preferences. Americans are more likely to dismiss AI risk post-hoc in order to justify making more money, whereas it seems that Chinese people are less likely to sacrifice their epistemic integrity in order to feel like a Good Guy, Hire people, and pay them money!

3Tomás B.2y

Pay Terry Tao his 10 million dollars!

3ChristianKl2y

If that's true I would assume that the people who work on creating the AI guidelines, understand the problem. This would in turn suggests that they take reasonable steps to address it. Is your model that the people writing the guidelines would be well-intentioned but lack the political power to actually enforce useful guidelines?

3Algon2y

Yeah, but a lot of people can say that w/o producing good work. There's been a number of complaints about field building attempts bringing in low quality people who don't make progress on the core problem. Now, I am NOT sayng you need to read the sequences and be a member of our cult to make progress. But the models in there do seem important to seeing what is hard about alignment. Now, many smart people have these models themselves, drawing from the same sources Yudkowsky did. But many smart people don't have these models and bounce off of alignment. I can sort of trust you that HPMOR is insanely cringe. I'm still not sure if a variant wouldn't work, because I don't have your model. Maybe I'll talk to some Chinese friends about this and get their opinion. You may be living in a bubble and not realize it. It happens to everyone at some point, and China must have a lot of bubbles.

3M. Y. Zuo2y

The private sentiment of folks who read through all of it would probably be some degree of 'cringe' too. I couldn't even make it halfway, though I am fascinated by imaginative fanfiction, as it becomes too much of a childish power fantasy to ignore and suspend my disbelief while reading.

5Algon2y

Yeah, you've got a point there. And yet, HPMOR is popular. Lots of people love it, and got into the LW and the rat community that way. You yourself may not have, but that's evidence in favour of high variance. So I remain unsure if something like HPMOR could work in China too. Why assume there'd be less variance in response there?

7astridain2y

It's about trade-offs. HPMOR/an equally cringey analogue will attract a certain sector of weird people into the community who can then be redirected towards A.I. stuff — but it will repel a majority of novices because it "taints" the A.I. stuff with cringiness by association. This is a reasonable trade-off if: 1. the kind of weird people who'll get into HPMOR are also the kind of weird people who'd be useful to A.I. safety; 2. the normies were already likely to dismiss the A.I. stuff with or without the added load of cringe. In the West, 1. is true because there's a strong association between techy people and niche fandom, so even though weird nerds are a minority, they might represent a substantial fraction of the people you want to reach. And 2. is kind of true for a related reason, which is that "nerds" are viewed as generally cringe even if they don't specifically talk about HP fanfiction; it's already assumed that someone who thinks about computers all days is probably the kind of cringe who'd be big into a semi-self-insert HP fanfiction. But in China, from @Lao Mein's testimony, 1. is definitely not true (a lot of the people we want to reach would be on Team "this sounds weird and cringe, I'm not touching it") and 2. is possibly not true (if computer experts ≠ fandom nerds in Chinese popular consciousness, it may be easier to get broad audiences to listen to a non-nerdy computer expert talking about A.I.).

-1Meena Kumar2y

HPMOR is weird and attracts weird people.

6Meena Kumar2y

Yudkowsky is not the right person to start this stuff in China.

3hairyfigment2y

All of that makes sense except the inclusion of "EA," which sounds backwards. I highly doubt Chinese people object to the idea of doing good for the community, so why would they object to helping people do more good, according to our best knowledge?

9Lao Mein2y

Yes. We hold volunteering in contempt.

0hairyfigment2y

See, that makes it sound like my initial response to the OP was basically right, and you don't understand the argument being made here. At least one Western reading of these new guidelines was that, if they meant anything, then the bureaucratic obstacle they posed for AGI would greatly reduce the threat thereof. This wouldn't matter if people were happy to show initiative - but if everyone involved thinks volunteering is stupid, then whose job is it to make sure the official rules against a competitive AI project won't stop it from going forward? What does that person reliably get for doing the job?

9Lao Mein2y

Volunteering to work extra hard at your job and break things is highly valued (and rewarded). Volunteering at your local charity is childishly naive. If your labor/time/intelligence was truly worth anything, you wouldn't be giving it away for free.

5Raemon2y

Hmm. It’s noteworthy that EA also thinks volunteering at your local charity is naive. EA is often trying to market itself in the west to the sort of people who by default might donate to local charity, but I think many of the ideas could be reworked-in-vibe substantially. (I guess the college student wanting to change the world being thought of as naive is more centrally EA though) That all said, I basically don’t think EA is necessary to make the case for AI, but I do think you need above average rationality to do real alignment work.

2Lao Mein2y

Donating to charity (regardless of effectiveness) is also viewed as naive. There is low empathy for poor people since many of us remember what it was like to have a much lower salary just a few years ago. I had to cut down on eating out and alcohol in order to live on $12,000 a year, but overall it was fine. I could still afford good food and expensive hobbies. There was a time when my entire family lived on a fraction of that, and it was completely fine. So why should I donate my hard-earned money to others who are living better than that?

6Richard_Kennaway2y

I am curious to know how Peter Singer is viewed in China (or would be, if he isn't yet known there). How would this talk go down with a Chinese audience? (The video isn't playing for me, but there is a transcript.) The thing is, while I reject his entire framework, I have never seen anyone give an argument against, to argue that supererogation is a thing, moral duty is bounded and is owed first to those nearest to you, having more than someone else does not imply a duty to give the excess away, and so on. These things are said, but never in my reading more than asserted.

[-]ChristianKl2y142

When it comes to helping strangers, the Chinese help strangers a lot less. There are articles like: https://medium.com/shanghai-living/4-31-why-people-would-usually-not-help-you-in-an-accident-in-china-c50972e28a82 about Chinese not helping strangers who have an accident.

When it comes to the accident that Peter Singer refers to, the article talks about it as:

There were a couple of extreme cases where Chinese people refusing to help led to the death of the person in need. Such is the case of Wang Yue, a two year old girl who was wandering alone in a narrow alley, because her mother was busy doing the laundry. She was ran over by two cars, and 18 people who passed around the area didn’t even look at her. Later a public anonymous survey revealed that 71% thought that the people who passed by didn’t stop to help her because they were afraid of getting into trouble.

The article goes on to say:

That’s not the only case in 2011 an 88 year old Chinese man fell in the street and broke his nose, while people passed him by no one helped him and he died suffocating in his own blood. After some anonymous poll the result was the same, the people didn’t blame those who didn’t help, because the rece

... (read more)

[-]Algon2y122

An example of the dangers of helping in China from the article that ChristianKl talked about.

While individualism in China is a big thing, this situation is more related to the fear of being accused as the responsible of the accident, even when you just tried to help.
The most popular case happened in the city of Nanjing, a city located at the west of Shanghai. The year was 2006 when Xu Shoulan, an old lady trying to get out of a bus, fell and broke her femur. Peng Yu, was passing by and helped her taking her to the hospital and giving her ¥200 (~30 USD) to pay for her treatment. After the first diagnosis Xu needed a femur replacement surgery, but she refused to pay it by herself so she demanded Peng to pay for it, as he was the responsible of the accident according to her. She sued him and after six months she won and Peng needed to cover all the medical expenses of the old lady. The court stated that “no one would, in good conscience, help someone unless they felt guilty”.
While this incident wasn’t the first, it was very popular and it showed one of the non written rules of China. [I bolded this] If you help someone it’s because you feel guilty of what happened, so in some way you

... (read more)

4ChristianKl2y

Historically, the problem seems to be that most Communist government initiatives to get people to be more altruistic result in them LARPing and not being more altruistic. At the moment the Chinese government solution seems to be: "Give everybody social credit scores that measure how altruistic they are and hopefully that will get everyone to be more altruistic".

4Algon2y

I don't get how larping altruism results in the norm that "if you're helping, you're guilty". Unless people went out of their way to cause problems and act like saviors to seem altruistic? Which might be possible, but also sounds like it would be difficult to execute and isn't the only way people could goodhart altruism metrics.

5ChristianKl2y

If most altruism is larping then for someone who does something altruistic, people who do altruistic things become suspect. If I remember right there are cases where in former communist countries people who always vote cooperate on the ultimatum game get punished for it. Communism isn't the only factor here but I would expect that it's one meaningful factor.

7Lao Mein2y

"OK, I guess I'm evil then."

2Raemon2y

Fair 'nuff.

1[comment deleted]2y

[-]1428572y194

A "moonshot idea" I saw brought up is getting Yudkowsky's Harry Potter fanfiction translated into Chinese (please never ever do this).

This has already been done, and has pretty good reviews and some discussions.

I've looked through the EA/Rationalist/AI Safety forums in China

If these are public, could you post the links to them?

there is only one group doing technical alignment work in China

Do you know the name of the group, and what kinds of approaches they are taking toward technical alignment?

6Lao Mein2y

Tian-xia forums are invite-only and mostly expats. I should probably dig deeper to find native Chinese discussions. CSAGI. Unfortunately, their website (csagi.org) has been dead for a while. It's founded by Zhu Xiaohu. He mentioned bisimulation and reinforcement learning.

9Zhu Xiaohu2y

Hi. Thanks for mentioning us. Unlike main labs or companies in China, we are doing fundamental research work on the ontological crisis problem with model theory from mathematical logic trying to set a new base for analyzing and preventing the crisis. Due to our lacking of funding and restricted intellectual resources, the process is slower, but we will share our work when ready.

4Lao Mein2y

Have you asked China EA people for funding? They mentioned that they would be interested in funding you.

2Zhu Xiaohu2mo

https://www.lesswrong.com/posts/3eAstSeZthYh2o6sh/ai-safety-through-operational-physics-why-resource?utm_campaign=post_share&utm_source=link

[-]johnswentworth2y184

This seems to be arguing against a starry-eyed idealist case for an "AI disarmament treaty", but not really against a cynical/realistic case. (At first I was going to say "arguing against a strawman", but no, there are in fact lots of starry-eyed idealists in alignment.)

Here's my cynical/realistic case for an "AI disarmament treaty" (or something vaguely in that cluster) with China. As the post notes, the regulations mostly provide evidence that Beijing sees near-term AI as a potential threat to stability that needs to be addressed with regulation. For purposes of an AI treaty, that's plausibly all we need. Near-term AI is a potential threat to stability from the CCP's perspective. That's true whether the AI is built in China or somewhere else; American-built AI is still a threat to stability. So presumably the CCP would love for the US to adopt rules limiting new LLMs. If the US comes along and says "we'll block training of big new AIs if you do", the CCP's answer is plausibly "yes definitely that sounds excellent".

And sure, China won't be working much on AI safety. That's fine; the point of an "AI disarmament treaty" is not to get literally everyone working on safety. They don't even need to be motivated by X-risk. If they're willing to commit to not build big new AI, then it doesn't really matter whether they're doing that for the same reasons we want it.

3Lao Mein2y

Parity in AI isn't what China is after - China doesn't want to preserve the status quo. We want to win. We want AI hegemony. We want to be years ahead of the US in terms of our AIs. And frankly, we're not that far behind - the recent Baidu LLMs perform somewhere between GPT2 and GPT3. To tie is to lose. Stopping the race now is the same as losing. I also don't see how LLMs can destabilize China in the near-term. Spam/propaganda isn't a big issue since you need to submit your real-life ID in order to post on Chinese sites.

3ChristianKl2y

If that's true, how do you explain the proposed guidelines that make it harder to train big models in China? The proposed guidelines suggest that the Cyberspace Administration of China believes that there are reasons that warrant slowing down LLM progress in China.

[-]Chris_Leong2y165

If only we could spread the meme of irresponsible Western powers charging head-first into building AGI without thinking through the consequences and how wise the Chinese regulation is in contrast.

I think it'll be easier in a few years to demonstrate the foolishness of open-source AI.

7Erich_Grunewald2y

That sort of strategy seems like it could easily backfire, where people only pick up the first part of that statement ("irresponsible Western powers charging head-first into building AGI") and think "oh, that means we need to speed up". Or maybe that's what you mean by "if only" -- that it's hard to spread even weakly nuanced messages?

2mako yass2y

I'd be worried about the meme reaching China, venting off any pressure they might have been under to actually develop solid regulation? Why would a politician drive for better regulation, and wouldn't it be a lot harder, if american AI people are saying they already trust you?

[-]ryan_b2y11-3

Since we are talking regulation and/or treaties, both government concerns, is there anything to the idea of working in the context of party slogans (by which I mean tifa [提法])?

What I have in mind is something like this:

Review the arguments about AGI in English
Look for conceptual overlap with slogans on various topics (I expect these to be national development, national security, and the socialist conception of historical laws, by way of example).
See if any can be updated gracefully to talk about AGI specifically.

I think this might work because:

It will sever AGI from EA/Rationalism
It will put the AGI arguments in a context more familiar to Chinese officials and party members at least

It will not work if:

Making obvious adjustments to slogans in order to associate a specific issue with it is culturally nonsensical, or viewed as presumptuous or otherwise low-status
It literally does not work linguistically in a "For make benefit glorious nation of Kazakstan" sort of way.

That being said, I think it might be a good exercise at least from the English side as a direct attempt at working with some Chinese conceptualization of the motivations for the problem of alignment. A few examples of wha... (read more)

6Algon2y

My worry about this is the same as my worry about making AI not-kill-everyone-ism palatable to politicians in the West. Will they get the actual, core problem, and so understand which policies will not help? Reading your suggested changes, I might pattern match AGI to a new and powerful nation. This interpretation has an implicitly human framing, viewing the advent of AGI is like the birth of a new hegemon. EDIT: That is to say, I'm not optimistic about confronting the mind-killing nature of political discourse, and trying to engage in that sort of discourse seems like a very hard battle. I am aware that I'm not giving a practical alternative though. So keep on thinking about this topic, please.

1ryan_b2y

My empathy runs deep. Providing a little more context for the sloganeering as done inside China's Communist Party, this is different from slogans in the context of American or European politics (which are almost totally tied to political campaigns for election); instead they work like this: Which of course does not resolve the problem of political competition: But I do have the sense that, at least in the case of party slogans, it is about what the priorities are and who executes them and the detailed implementation is usually a separate question.

-2Algon2y

I don't understand why this helps. Who executes a priority, and what exactly a priority is, seem greatly correlated with the space of detailed implementation of a policy. Look at what happened with Drexlerian nanotech: the term got hijacked by people who called their pre-existing work nanotech in order to obtain resources from the US government which were earmarked for "nanotech". Why wouldn't something similair happen for AI not-kill-everyoneism? People argue over what exactly the priority is ("the AI must have chinese characteristics" vs. "the AI must be rewarded for having chinese characterisitcs and obeying the law") and who executes it (curious, brilliant people who can work on the core of the problem vs bureaucrat clout-chasers). So what if the detailed implementaion is a seperate question? The front has already collapsed. I admit to that I don't see what has made you excited about this idea, and understand if you don't want to spend the effort conveying it at the moment. And I also admit to being confused: I realized that part of where the nanotech-AI analogy might fail is in the pressures US vs. Chinese politicians face, and how the battle over priorities are fought. Another area it might fail is that I don't in what context "sloganeering" is done. Who is the audience for this? How does the existence of a dictator like Xi affect things? I've not really thought about it.

4ryan_b2y

This is the crux of the matter, I think: the slogans to which I am pointing are those used inside the communist party of China for the purposes of coordinating the party members and bureaucrats, who are the audience. Xi has introduced several of the slogans in current use, and has tried and failed to introduce others. That is to say, they are how the Chinese government talks to itself, and Xi is at the center of the conversation. I focused on the slogans because I have some clue how this system works, but don't have a notion about Chinese language in general, or Chinese culture in general, or the technical culture specifically. So all I've done here is take the idea "alignment should be more of a priority in China" and the idea "I know one way the Chinese government talks about priorities" and bashed 'em together like a toddler making their dolls kiss. The challenge is the part that is exciting to me, frankly. Communicating an important problem across cultural lines is hard, and impressive when done well, and provides me a certain aesthetic pleasure. It is definitely not the case that I have analyzed the problem at length, or done similar things before and concluded on priors that this will be an effective method. Edit: putting the slogans into a more LessWrong context, tifa are directly a solution to the problem described n You Get About Five Words.

3Lao Mein2y

Party slogans are LARP. Westerners trying to convince us of things LARPing with our LARP is... Actually, just don't do it.

4ryan_b2y

How do ideas usually enter the party sphere, out of curiosity?

6Lao Mein2y

Expert opinion, conversations with their engineer friends, the "general mood" among scientists in a particular field, and opinion pieces written by influential writers. That sort of thing.

[-]otto.barten2y101

Me and @Roman_Yampolskiy published a piece on AI xrisk in a Chinese academic newspaper: http://www.cssn.cn/skgz/bwyc/202303/t20230306_5601326.shtml

We were approached after our piece in Time and asked to write for them (we also gave quotes for another provincial newspaper). I have the impression (I've also lived and worked in China) that leading Chinese decision makers and intellectuals (or perhaps their children) read Western news sources like Time, NYTimes, Economist, etc. AI xrisk is currently probably mostly unknown in China, and if stumbled upon people might have trouble believing it (as they have in the west). But if/when we're going to have a real conversation about AI xrisk in the west, I think the information will seep into China as well, and I'm somewhat hopeful that if this happens, it could perhaps prepare China for cooperation to reduce xrisk. In the end, no one wants to die.

Curious about your takes though, I'm of course not Chinese. Thanks for the write-up!

[-]RobertM2y72

I've heard people be somewhat optimistic about this AI guideline from China. They think that this means Beijing is willing to participate in an AI disarmament treaty due to concerns over AI risk.

I'm curious where you've seen this. My impression from reading the takes of people working on the governance side of things is that this is mostly being interpreted as a positive sign because it (hopefully) relaxes race dynamics in the US. "Oh, look, we don't even need to try all that hard, no need to rush to the finish line." I haven't seen anyone seri... (read more)

[-]Lao Mein2y1912

I've only seen vaguely positive vibes from people who needed Google Translate in order to understand it, like this post from Zvi. He comes to the conclusion that "All of that points, once again, to an eager partner in a negotiation." This isn't obvious at all. Again, a willingness to regulate nuclear power is not a strong signal for a willingness to participate in nuclear disarmament treaty - all states will eventually have some level of nuclear regulation.

[-]jacob_cannell2y60

Thanks, I'm now more curious about chinese viewpoints on AI and the singularity in general.

[-]Lao Mein2y233

"Damn it, we're falling behind! GPT4 is way better than anything we have."

OpenAI's bans on Chinese users have really hurt public knowledge of GPT4, for what that's worth. The small amount of effort it takes to get a US phone number was enough to discourage casuals from getting hands-on experience, although there are now clone websites using an OpenAI API going up and down all the time. But yeah, awareness of just how good it is isn't as mainstream compared to the US.

As far as I can tell, GPT4/ChatGPT works great on Chinese, even without fine-tuning. And it blows Chinese-specializing models of Baidu and friends out of the water. It seems like a bit of a Spudnik moment.

8jacob_cannell2y

Interesting - OpenAI is actually banning chinese users, or is the great firewall banning OpenAI? I can't find a quick answer from google, instead I"m getting reports that its a mutual ban/restriction? I can't immediately see why it would be in OpenAI's interest to not allow at least paying chinese users.

[-]Lao Mein2y210

OpenAI does not accept creating an account with a Chinese/HK phone number (specifically noting that they are blocking service from those areas). I also can not access the chatgpt website from a mainland IP, but can from HK. I vaguely remember OpenAI citing US law as a reason they don't allow Chinese users access, maybe legislation passed as part of the chip ban?

1Erich_Grunewald2y

Nah, the export controls don't cover this sort of thing. They just cover chips, devices that contain chips (i.e. GPUs and AI ASICs), and equipment/materials/software/information used to make those. (I don't know the actual reason for OpenAI's not allowing Chinese customers, though.)

6M. Y. Zuo2y

Any online service hosted outside of mainland China wanting to sell within has to meet quite onerous regulations. I've thought about for some time why these exist and develop to the extent that it's harmful (and why they exist nearly everywhere). It's partly by design, similar to the EU's regulatory apparatus that created GDPR, partly because of the natural distrust of government officials towards the intentions of foreign actors, but also because of the office dynamics. As scoring points against foreigners is low hanging fruit for the career prospects of middle-management government officials in many, many, government offices. If their attempts succeed in delivering proof of misdeeds, or a public apology, then it's a guaranteed promotion. If their attempts fail, how will the overseas stakeholders retaliate against these nameless official(s) without pointing the finger at the department/division/ministry/government/China... as a whole? If they do anyways, it would turn into a status fight, thus facilitating a promotion for whoever proposed it. And then some stricter regulations will be imposed as retaliation, which increases the power and authority of the instigators. i.e. In either case the instigator wins, so it becomes a ratcheting mechanism that encourages ever more extreme proposals and tit-for-tat behaviour, and that's even before the geopolitics come into play In this sense increasing geopolitical tension might ironically reduce the day-to-day risk of being on the unfortunate end of such schemes, since higher level officials will be extra motivated to make sure their subordinates are more disciplined.

[-]Jordan Schneider2y51

Hi! I run a reasonably popular podcast (ChinaTalk) and am also very frustrated with the misconceptions people have around this topic! Pls reach out I’m at jordan@chinatalk.media

[-]Mitchell_Porter2y50

Thanks very much for this dose of reality. So maybe a western analogy for the attitude to "AI safety" at Chinese companies, is that at first, it will be comparable to the attitude at Meta. What I mean by this: Microsoft works with OpenAI, and Google works with Anthropic, so they both work with organizations that at least talk about the danger of AI takeover, alongside more mundane concerns. But as far as I can tell, Meta does not officially acknowledge AI takeover as a real risk at all. The closest thing to an official Meta policy on the risk of AI takeove... (read more)

[-]mic2y30

Relevant: China-related AI safety and governance paths - Career review (80000hours.org)

[-]mako yass2y2-1

Do you really think that a scientist is going to walk up to his friend from the Politburo and say "Hey, I know AI is a central priority of ours, but there are a few fringe scientists in the US asking for treaties limiting AI, right as they are doing their hardest to cripple our own AI development. Yes, I believe they are acting in good faith"

This part focused me, what happens if you do this is, your politburo friend answers "that's obviously dangerous propaganda, so I'm going to generate counterarguments against it so that no one entertains it ever again, ... (read more)

[-]ChristianKl2y20

It is far more likely that Beijing sees near-term AI as a potential threat to stability that needs to be addressed with regulation.

This suggests that Beijing is making regulations that are driven by short-term thinking and not careful thinking over longer timelines.

Generally, my perspective of Chinese decision-making is that government policy is written by people who are able to think over longer timelines.

To me "AGI has to be aligned with CEV" and the sentence of "AGI has to be aligned with socialist values" sound very similar to me. Anything ... (read more)

3Lao Mein2y

Chinese experts simply don't think it's a real issue yet. There is no law of physics that says our long-term assessment has to match that of the West. Just because you think it's in our best interest doesn't mean we have to.

2ChristianKl2y

Do we have a good reason to assume that the people at the Cyberspace Administration of China are open and transparent about their long-term assessments? What kind of experts do you think would say something different publically if such assessments play into their policy?

[-]GeneSmith2y20

Let us know when the podcast with you goes up!

[-]ChristianKl2y20

When it comes to having an international treaty, whether or not China will agree depends a lot on the content of the treaty.

If the Chinese put their guidelines in practice, they might be open to have a treaty that basically says that all companies have to abide by the guidelines.

The proposed requirements on training data quality can prevent models like GPT-3 and GPT-4 from being trained.

Legal liability for anything the model does is also an effective way to slow down AI development.

[-]ShenZhen2y20

Old lurker new account! Need to go work soon so very very quick high-level thoughts:

I feel like we feel some of the same frustration at mainstream/western EA, or others alignment speculaters without a deep understanding of China. I can crudely gesture at their wrong inferences sourcing from assumptions orthogonal to the truth, implying fundamental misconceptions in their model, but in the end it comes down to context: much of this is sub-verbal subtle differences that I can succinctly identify without putting in a few months of effort first. Often when I t... (read more)

[-]NirViaje1y10

我浏览了中国的EA/Rationalist/AI安全论坛

Hi, do you know Concordia AI in Beijing? they are pretty active -
https://concordia-ai.com/
https://www.zhihu.com/org/an-yuan-ai-6/posts

[-]Review Bot2y*10

The LessWrong Review runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2024. The top fifty or so posts are featured prominently on the site throughout the year.

Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?

[-]Vael Gates2y*10

Does Anyuan(安远) have a website? I haven't heard of them and am curious. (I've heard of Concordia Consulting https://concordia-consulting.com/ and Tianxia https://www.tian-xia.com/.)

2Lao Mein2y

My mistake, 安远 is Concordia's Chinese name.

[-]angmoh2y10

Good post.

For Westerners looking to get a palatable foothold on the priorities and viewpoints of the CCP (and Xi), I endorse "The Avoidable War" written last year by Kevin Rudd (former Prime Minister of Australia, speaks mandarin, has worked in China, loads of diplomatic experience - imo about as good of a person that exists to interpret Chinese grand strategy and explain it from a Western viewpoint). The book is (imo, for a politician), impressively objective in its analysis.

Some good stuff in there explaining the nature of Chinese cynicism about foreign motivations that echoes some of what is written in this post, but with a bit more historical background and strategic context.

[-]Joshua Clancy2y10

I did a masters of data science at Tsinghua University in Beijing. Maybe it's a little biased, but I thought they knew their stuff. Very math heavy. At the time (2020), the entire department seemed to think graph networks, with graph based convolutions and attention was the way forward towards advance AI. I still think this is a reasonable thought. No mention of AI safety, though I did not know about the community (or concern) then.

[-]Meena Kumar2y10

The main thing seems to be money. How do we get more money into alignment in a way that is not detrimental or repulsive to people we want to attract to working there?

[-]Ariel_2y10

I'm trying to think of ideas here. As a recap of what I think the post says:

China is still very much in the race
There is very little AI safety activities there, possibly due to a lack of reception of EA ideas.

^let me know if I am understanding correctly.

Some ideas/thoughts:

it seems to me that many in AI Safety or in other specific "cause areas" are already dissociating from EA, though not much from LW.
I am not sure if we should expect mainstream adoption of AI Safety ideas (its not really mainstream in the west, nor is EA, or LW).
It seems like the

... (read more)

4Lao Mein2y

I'm saying that AI Safety really can't take off as a field in China without a cadre of well-paid people working on technical alignment. If people were working on, say, interpretability work here for a high wage ($50,000-$100,000 is considered a lot for a PHD in the field), it would gain prestige and people would take it seriously. Otherwise it just sounds like LARP. That's how you do field building in China. You don't go around making speeches, you hire people. My gut feeling is that hiring 10 expats in Beijing to do "field building" gets less field building done than hiring 10 college grads in Shanghai to do technical alignment work.

3Lichdar2y

My experience is that the Chinese(I am one) will disassociate with the "strange part" of EA, such as mental uploading or minimization of suffering or even life extension: the basic conservative argument for species extension and life as human beings is what works. CN is fundamentally conservative in that sense. The complications are not many and largely revolve around: 1. How is this good for the Party. 2. How is this good for "keeping things in harmony/close to nature/Taoism" 3. Emotional appeals for safety and effort. The story of effort = worth is strong there, and devaluation of human effort leads to reactions of disgust.

2Lao Mein2y

Mind uploading and life extension are far easier sells than ending factory farming, in my experience. It's not the tech part we find cringe, but the altruism part. The Chinese soul wants to make money, be cool, and work in a cool field. People trying to sell me altruism are probably part of a doomsday cult (e.g. Catholicism).

2Lichdar2y

I think the lack of altruism part comes from the desire to compete and be superior(via effort). There's no vast desire to have altruism as the West understands it. How would you be better than others? The upside is that "specism" is a norm. There would be zero worry about wanting to give AI "fair rights." I do think there is some consideration for "humankind", but individual rights(for humans or animals) are strictly a "nice to have, unnecessary and often harmful."

[-]Victor Lecomte2y*10

It sounds like you're skeptical about EA field building because most Chinese people find "changing the world" childishly naive and impractical. Do you think object-level x-risk field building is equally doomed?

For example, if 看理想 (an offshoot of publishing house 理想国 that produces audio programs about culture and society) came out with an audio program about x-risks (say, 1-2 episodes about each of nuclear war, climate change, AI safety, biosecurity), I think the audience would be fairly receptive to it. 梁文道, one of the leaders of 看理想, has shared on his pod... (read more)

[-]Lao Mein2y105

We know we can change the world - it's not even a question when your grandparents are literal peasant farmers, your parents are industrial workers, and you're a scientist. It's just that altruism itself is cringe. Volunteering is cringe. I don't know how to explain this to Westerners in a way they can understand.

3Lichdar2y

For Westerners, I think the explanation can go like this. Say you want to do an open source thing, and spend hours on it. Okay. But if you are in China, you could have used those hours to make money for your family. Why would you take care of not-family unless you had too much money and time? I do think the Chinese care about "humanity" in my experience, but Three Body Problem shows a typical version of it: survival of the mass via individual cruelties. Like I said, though, I think the upside is that the Chinese are naturally unreceptive to anything like rights for AI.

1Victor Lecomte2y

I would still like to try and understand, if that's okay. :) Would you say the following captures some of it?

5Lao Mein2y

That's pretty close to it. And your parents and teachers regularly take you aside and remind you that it's all LARP. I also suspect that the desire to engage in charitable behavior and political organization is motivated by cultural neuroses. Why should you go vote or politically organize or give your time away for free when you can so obviously improve your life by putting effort into your career? Christian/liberal guilt, I assume.

LESSWRONG
LW

LESSWRONG
LW

252

My Assessment of the Chinese AI Safety Community

252

252