https://arxiv.org/abs/2107.10939 might be useful reading:
We describe cases where real recommender systems were modified in the service of various human values such as diversity, fairness, well-being, time well spent, and factual accuracy. From this we identify the current practice of values engineering: the creation of classifiers from human-created data with value-based labels. This has worked in practice for a variety of issues, but problems are addressed one at a time, and users and other stakeholders have seldom been involved. Instead, we look to AI alignment work for approaches that could learn complex values directly from stakeholders, and identify four major directions: useful measures of alignment, participatory design and operation, interactive value learning, and informed deliberative judgments.
Suppose you are not a superhuman AGI, but instead a startup in 2006 (destined to grow to a large global company by 2022).
(This setup hopefully makes it less complicated then a Open-Source Wish Project as we use a company of smart young people, not an omnipotent mean genie, to steer the story towards disaster)
Your product is a social network.
If you optimize for money, in 2022 you'll sell targeting ads via psychological profiles of your users, which is a bad outcome.
If you optimize for user engagement, in 2022 you'll be selling in-game virtual goods to addicted mobile gamers escaping from sad reality, which is a bad outcome.
If you optimize for the number of social connections, in 2022 you'll end up with ...realistically with no product anymore, as people will abandon a place where there are 1000 fake accounts of Selena Gomez, Santa Claus and Pope created just to collect the most number of edges with no real human connection (which I happen to know firsthand) - which is a bad outcome.
If you optimize for the number of photos, posts, or videos uploaded, you'll probably get swamped in low quality content.
If you optimize for the quality of the content measured by number of likes, you'll probably end up creating a culture of low esteem people seeking external validation nervously refreshing notification tab waiting for the slot machine to show them love. Which is a bad outcome.
If you optimize for quality as in amount of time spent consuming content you'll probably get bing watching and/or content marketing ranging from benign product placement to distorting reality in favor of sponsor's views.
If you optimize for quality measured by number of clicks or shares you'll end up with click-bites and, click-stealing iframes, "tell your friends!" popups and swamped notifications.
If you optimize for the amount of interactions between members, you'll probably get spam.
If you optimize for two-way interactions, then you'll probably get culture wars.
If you optimize for number of unique people with which a user hold at least an hour long conversation with at most 100 message exchanges and generally positive sentiment, you'll get ... adult sex chat?
What should be the Key Performance Idicitactors of the Social Network developed in 2006 which you'd like to see in 2022?