TL;DR: Access to AI safety talent & resources, particularly for labs during training, is sorely lacking in China. We created one of the largest AI safety communities in China, and plan to next start with auditing reward-hacks of unreliable upstream RL environments that domestic data vendors provide to Chinese frontier labs (which induces reward hacking and a whole host of dangerous downstream tendencies).
RL tasks are notoriously prone to exploitation by reward-seeking agents (see examples here, here, and here). In Xiaomi's MiMo v2.6 RL environments that were open-sourced, 2/3 had leaks on Git logs. There's a few clusters of common pitfalls that environments run into, but overall things evolve quickly. From our conversations with employees, CN labs typically don't divert enough people to this particular area, especially since project leads would want senior researchers to work on capabilities research, and because slowing down the development cycle is widely seen as too costly.
The rough shape of our initial solution is an embedded team of staff directly working on identifying & fixing broken environments, working in tandem with the alignment & safety departments within each lab (afaik we estimate that the main Chinese AI players only have 5-10 people per company working on alignment and harm reduction). We are currently in good standing with several labs (our courses' students are often lab employees).
We also plan to produce seasonal (and case-by-case) public reports of specific reward hacking failure modes that we are seeing in RL environments, as well as practical recommendations. This will benefit independent researchers, other evaluators, and data vendors in terms of insights into reducing such failure modes and reducing overall harm. We'd also like to participate in drafting future national and international standards of safety during the training process of AIs.
This is considered by us as a near-term harm reduction strategy for safety & alignment, much like Erowid is to drug safety. It's addressing one of the most pressing current issues that we are seeing direct interest from domestic labs. However, we do have more ambitious plans to expand to areas that more directly address automated alignment in the near future, especially as Chinese companies and the state gradually notice the severity of the issue with near-future warning shots.
We are an ambitious but young team and would love to hear feedback and explain more in DMs - especially from more experienced researchers in the West. Research partnerships, investments/donations, or just striking up a casual conversation are all appreciated!
TL;DR:
Access to AI safety talent & resources, particularly for labs during training, is sorely lacking in China. We created one of the largest AI safety communities in China, and plan to next start with auditing reward-hacks of unreliable upstream RL environments that domestic data vendors provide to Chinese frontier labs (which induces reward hacking and a whole host of dangerous downstream tendencies).
RL tasks are notoriously prone to exploitation by reward-seeking agents (see examples here, here, and here). In Xiaomi's MiMo v2.6 RL environments that were open-sourced, 2/3 had leaks on Git logs. There's a few clusters of common pitfalls that environments run into, but overall things evolve quickly. From our conversations with employees, CN labs typically don't divert enough people to this particular area, especially since project leads would want senior researchers to work on capabilities research, and because slowing down the development cycle is widely seen as too costly.
The rough shape of our initial solution is an embedded team of staff directly working on identifying & fixing broken environments, working in tandem with the alignment & safety departments within each lab (afaik we estimate that the main Chinese AI players only have 5-10 people per company working on alignment and harm reduction). We are currently in good standing with several labs (our courses' students are often lab employees).
We also plan to produce seasonal (and case-by-case) public reports of specific reward hacking failure modes that we are seeing in RL environments, as well as practical recommendations. This will benefit independent researchers, other evaluators, and data vendors in terms of insights into reducing such failure modes and reducing overall harm. We'd also like to participate in drafting future national and international standards of safety during the training process of AIs.
This is considered by us as a near-term harm reduction strategy for safety & alignment, much like Erowid is to drug safety. It's addressing one of the most pressing current issues that we are seeing direct interest from domestic labs. However, we do have more ambitious plans to expand to areas that more directly address automated alignment in the near future, especially as Chinese companies and the state gradually notice the severity of the issue with near-future warning shots.
We are an ambitious but young team and would love to hear feedback and explain more in DMs - especially from more experienced researchers in the West. Research partnerships, investments/donations, or just striking up a casual conversation are all appreciated!
Emails:
cmlkevinlin@gmail.com
yuanyuan.sun.sarah@gmail.com