I think these are great points. Entirely possible that a really good appropriately censored LLM becomes a big part of China's public-facing internet.
On the article about Iowa schools, I looked into this a little bit while writing this and as far as I could see rather than running GPT over the full text and asking about the content like what I was approximating, they are instead literally just prompting it with "Does [book X] contain a sex scene?" and taking the first completion as the truth. This to me seems like not a very good way of determining whether books contain objectionable content, but is evidence that bureaucratic organs like outsourcing decisions to opaque knowledge-producers like LLMs whether or not they are effective.
Feel free to DM. I think you're absolutely correct these systems will eventually be used by intelligence agencies and other parts of the security apparatus for fine-grained targeting and espionage, as well as larger scale control mechanisms if they have the right data. This was just the simplest use of the current technology, and it seems interesting that mass monitoring has still been somewhat labor-constrained but may not remain so. These sorts of immediate concerns may also be useful for better outreach in governance/policy discussions.
This was a post I wrote during SERI MATS and not my main research. Some of the folks working on hacking and security are more explicitly investigating the potential of targeted operations with LLMs.
I think you're probably right, my feeling is that organic pro-regime internet campaigns are possibly more important than traditional censorship. The PRC has been good at this and I've also been worried about how vocal Hindutva elements are becoming.
I don't know that we've yet found the optimal formula for information control (which is a good thing) and I remain a little agnostic on the balance between censorship and propaganda. This post focused on old-style censorship because it's better documented, but a contemporary information control strategy necessarily involves a lot more.
I've so far been skeptical of a lot of misinformation narratives because I don't think fake news articles for example are labor constrained, but LLMs can definitely be used to boost in the official narrative in more interesting ways. Looking at the PRC again, at least some people in Xinjiang have reported being coerced into posting positively on social about state-narratives, and I have Chinese contacts who have been discouraged socially from posting negative things. I'm guessing some of the censorship tools can also be used to subtly encourage such behaviors and grow the pro-regime mobs.