As AI becomes more integrated into software, there’s a risk that it will exacerbate the flaws of existing tech and disempower humans. It might be useful for some people to work on software that would instead be specifically designed to empower human users and help them achieve their considered goals. Instead of software that creates maximally addicting content, we want software that allows us to create, connect, and express ourselves better. Instead of writing an essay or a song for me, AI should help me write it and realise my vision.
Even if we avoid accidents of the Hugging Face incident type, I fear that AI would be misaligned with us; because current AIs and current software are already misaligned with us. For example, YouTube’s recommendation algorithm is misaligned with me. I think it would be best for me to watch educational videos about things like psychology or AI, or if it’s night-time, something calming, perhaps a meditation video. But the algorithm is designed to keep me on the platform as long as possible, so it keeps recommending me basketball videos, and I can’t help clicking on them, even though it's not what I want to be doing.
I can imagine an AGI being misaligned to us in a similar way. Rather than rebelling directly against us, it might simply become extraordinarily good at giving us whatever we appear to want right now. It might remove friction, eliminate waiting, and satisfy every revealed preference. But this kind of power may itself be deeply misaligned with what humans actually want from their lives.
What would an aligned YouTube algorithm look like? It might allow us to specify what videos we want recommended and hidden and when. We could also specify our goals to the algorithm. E.g., “I want to be more in touch with my emotions and understand topics that are relevant for me professionally more deeply.” The algorithm could then ask how much various videos I watched contributed to these goals, and update accordingly. I’d love for someone to make such an app or website. It could be called MeTube or UsTube.[1]
It could also be useful to research what healthy and unhealthy AI use looks like, then use those findings to build LLMs that encourage healthier patterns of interaction. These systems could include guardrails against harmful or compulsive use, while actively supporting autonomy, reflection, learning, emotional awareness, and real-world engagement. This could potentially be built now on top of open source or open-weights models.
Given advances in AI and programming, it may now be possible to design operating systems around our considered goals rather than just our immediate impulses. They could limit doomscrolling or gaming across devices, restrict device use at certain times, and block distractions while we focus. Existing software can already do some of this, but an AI based OS could go further by actively helping us achieve the goals we set for ourselves, and stop us from using our devices in ways we don’t want to.
There are some risks with this sort of holistic system. People could use them to create filter bubbles of their own making, curating their content such that they are only exposed to ideas they already agree with, no matter how extreme. In this way, they could increase polarisation, isolation, and misinformation: for example, a dictator could make their own personalised OS which only allowed messages that they approved.
Another problem is that many of us don’t know what we truly want. Imagine Bob, a lawyer. He’s watching videos about basketball, but the ideal version of himself would instead watch law-related videos to advance his professional development. But perhaps Bob is unaware that in fact, he wants to advance in his legal career partly to prove to his dad that he is worthy. Maybe if he understood that and healed it, he would realise that he actually wants to be a gardener rather than a lawyer, and that pursuing a gardening career would ultimately make him much happier.
In a positive AI future, AI might somehow help us figure out these deepest desires, and then cater to them. But we’d need to trust that the AI truly understood and cared about these deep desires. If the AI was even slightly misaligned AI, it could end up manipulating and further disempowering humans.
Humans need to have the final word on what values and goals our technology helps us achieve: otherwise it would lead to disempowerment. The question is, can we actually build AI systems that help people to discover and act on their own considered values, without the systems influencing those values?
It could be somewhat important to work on this soon. As AI becomes more and more integrated into software, there could be path dependency. The first movers could set the stage, become popular and then pour more resources into optimisation, and it might then be difficult for new approaches to beat them. Therefore, we may currently be in an unusually important window, in which trends for AI-mediated software are still fluid. Early demonstrations of genuinely human-empowering software could disproportionately influence what later systems look like.
It might be better if these systems were open source. This will help adoption, and make them more customisable. If they’re for-profit, there is a risk that even well-meaning startups could become influenced by profit incentives over time. Lots of companies (like OpenAI) start out saying they want to create things for public benefit, but when they’re faced with a choice between profit and helping the world, they choose profit. A disadvantage of these systems being open source is that it also increases risks of dictators misusing them to create Dictator AI.
Ultimately, I am a lot more concerned about AIs destroying us than AI integrations being suboptimal, and I definitely don’t want to draw attention away from x-risk. But our software design might tend to disempower or empower people, and we can aim for empowerment.
Other ideas for MeTube: it could flag if you’re doing things you don’t want to be doing, e.g. binging on basketball videos at 1am. It could also allow people to specify or tag the way in which videos affected them, e.g. “deeply touching”, “informative”, “funny”, etc., and then serve them similar content when they want to moved, informed, or amused.
As AI becomes more integrated into software, there’s a risk that it will exacerbate the flaws of existing tech and disempower humans. It might be useful for some people to work on software that would instead be specifically designed to empower human users and help them achieve their considered goals. Instead of software that creates maximally addicting content, we want software that allows us to create, connect, and express ourselves better. Instead of writing an essay or a song for me, AI should help me write it and realise my vision.
Even if we avoid accidents of the Hugging Face incident type, I fear that AI would be misaligned with us; because current AIs and current software are already misaligned with us. For example, YouTube’s recommendation algorithm is misaligned with me. I think it would be best for me to watch educational videos about things like psychology or AI, or if it’s night-time, something calming, perhaps a meditation video. But the algorithm is designed to keep me on the platform as long as possible, so it keeps recommending me basketball videos, and I can’t help clicking on them, even though it's not what I want to be doing.
I can imagine an AGI being misaligned to us in a similar way. Rather than rebelling directly against us, it might simply become extraordinarily good at giving us whatever we appear to want right now. It might remove friction, eliminate waiting, and satisfy every revealed preference. But this kind of power may itself be deeply misaligned with what humans actually want from their lives.
What would an aligned YouTube algorithm look like? It might allow us to specify what videos we want recommended and hidden and when. We could also specify our goals to the algorithm. E.g., “I want to be more in touch with my emotions and understand topics that are relevant for me professionally more deeply.” The algorithm could then ask how much various videos I watched contributed to these goals, and update accordingly. I’d love for someone to make such an app or website. It could be called MeTube or UsTube.[1]
It could also be useful to research what healthy and unhealthy AI use looks like, then use those findings to build LLMs that encourage healthier patterns of interaction. These systems could include guardrails against harmful or compulsive use, while actively supporting autonomy, reflection, learning, emotional awareness, and real-world engagement. This could potentially be built now on top of open source or open-weights models.
Given advances in AI and programming, it may now be possible to design operating systems around our considered goals rather than just our immediate impulses. They could limit doomscrolling or gaming across devices, restrict device use at certain times, and block distractions while we focus. Existing software can already do some of this, but an AI based OS could go further by actively helping us achieve the goals we set for ourselves, and stop us from using our devices in ways we don’t want to.
There are some risks with this sort of holistic system. People could use them to create filter bubbles of their own making, curating their content such that they are only exposed to ideas they already agree with, no matter how extreme. In this way, they could increase polarisation, isolation, and misinformation: for example, a dictator could make their own personalised OS which only allowed messages that they approved.
Another problem is that many of us don’t know what we truly want. Imagine Bob, a lawyer. He’s watching videos about basketball, but the ideal version of himself would instead watch law-related videos to advance his professional development. But perhaps Bob is unaware that in fact, he wants to advance in his legal career partly to prove to his dad that he is worthy. Maybe if he understood that and healed it, he would realise that he actually wants to be a gardener rather than a lawyer, and that pursuing a gardening career would ultimately make him much happier.
In a positive AI future, AI might somehow help us figure out these deepest desires, and then cater to them. But we’d need to trust that the AI truly understood and cared about these deep desires. If the AI was even slightly misaligned AI, it could end up manipulating and further disempowering humans.
Humans need to have the final word on what values and goals our technology helps us achieve: otherwise it would lead to disempowerment. The question is, can we actually build AI systems that help people to discover and act on their own considered values, without the systems influencing those values?
It could be somewhat important to work on this soon. As AI becomes more and more integrated into software, there could be path dependency. The first movers could set the stage, become popular and then pour more resources into optimisation, and it might then be difficult for new approaches to beat them. Therefore, we may currently be in an unusually important window, in which trends for AI-mediated software are still fluid. Early demonstrations of genuinely human-empowering software could disproportionately influence what later systems look like.
It might be better if these systems were open source. This will help adoption, and make them more customisable. If they’re for-profit, there is a risk that even well-meaning startups could become influenced by profit incentives over time. Lots of companies (like OpenAI) start out saying they want to create things for public benefit, but when they’re faced with a choice between profit and helping the world, they choose profit. A disadvantage of these systems being open source is that it also increases risks of dictators misusing them to create Dictator AI.
Ultimately, I am a lot more concerned about AIs destroying us than AI integrations being suboptimal, and I definitely don’t want to draw attention away from x-risk. But our software design might tend to disempower or empower people, and we can aim for empowerment.
Thank you to Amber Dawn Ace for helping to turn my messy drafts into this text. Cross-posted on the EA forum.
Other ideas for MeTube: it could flag if you’re doing things you don’t want to be doing, e.g. binging on basketball videos at 1am. It could also allow people to specify or tag the way in which videos affected them, e.g. “deeply touching”, “informative”, “funny”, etc., and then serve them similar content when they want to moved, informed, or amused.