One part of it is whether the assistant is able and willing to interact with me in a way that is compatible with how I want to grow as a person.

I think this argument unfortunately undercuts the entire concept of Rationality, and for this reason I think it is a good argument. Not because it undercuts Rationality, but because it points to what I think is the underlying concern of all humane cultural systems attempting to allow humans to progress, namely "what is good, what is bad/what is right/wrong, what is true/false." But I'm not convinced that all things in the world are either Good or Bad. I personally believe developing the ability to lie, and to lie convincingly is a necessary skill for functioning as an effective adult, and I wonder myself how concepts like that are treated in a world of 1's and 0's. 

Consider a world where if we really believe in the binary idea that people are either Good or Bad though, and the logical follow up that Good action/thinking is good, and vice versa. Then in a simpler world, as rationalists in this world, if we are trying to do good, we are Good. This also means if we are trying to do bad, then we are Bad.

In this world, I would prefer to be on the side of Good given my current understanding of what Good means, which means if someone is trying to do bad but dress it up as Good, it is still Bad. A saying like "Wolves in sheeps clothing" sums this idea up decently.  Given their skill level though, things could still turn out good if the fake sheep are proven to be wolves (Bad). "Crime never pays" points this idea out.

Conversely, I think that if someone is trying to do Good, but dress it up as Bad, there is the potential for it to still be Good, but also depending on their skill level it could still be Bad if it cannot be revealed for being Good. Robin hood "Steals from the Rich and Gives to the Poor" but "The road to Hell is paved with good intentions" address these ideas respectively.

So in this world, if an AI is developed to align with what it's user wants, and the user wants to use it for Bad, then I think this AI is Bad and it's development should not be pursued. Period.

Another part of the question is whether people in general want to become more prosocial or more cunning, or whatever. Or if they even have coherent desires around this. 

This is where I think a binary approach to this problem is flawed, for reasons I've tried in very early draft form, to illustrate here. Basically I wonder if the pace of Binary Computing Technology has accelerated and influenced the development of human culture along the flawed idea that there are only true/false relationships in the universe, and done it so well, that this binary approach to thinking is crowding out other ways of thinking. I think of the Borg from Star Trek as simple cultural allusion. 

Psychology is the field of study that deals with ideas issues of pro-social/anti-social behavior and the ability for humans to rationally understand their desires and motivations. It's also a product of human culture and so the data and meaning it produces can also be used to argue for whatever it is that human culture wants it to mean at the time, past, present, and/or future. "Culture creates meaning" gets to this idea well.

Today, there are still different cultures around the world, despite Internet Culture fighting for World dominance at this point. This cultural struggle is the same human process that gets repeated over and over again, and it is superior technology which determines who will win, not objective ideas about what is truly good or bad, yet. This is the same warfare process that has resulted in so much pain, misery and suffering around the world since the dawn of warfare, and it is unlikely to change because human technological advancement has far outstripped human development. 

Regardless of how we want to dress it up, we are still passing around many of the same flawed ideas about the world, how it should be, and how it should be fixed. It's just being done soooooo much faster and is reaching across the entire globe and reaching into more and more facets of everyday life now. 

Moral ideas that collapse the Universe into what's Good and what's bad from Dominant Western Culture and Civilization, seem to translate very well into 1's and 0's philosophically. 

So despite the efforts of a lot of smart and well meaning people, Internet culture continues to spread a Dominant Western Culture based on Judeo Christian Values around the world, attempting to define and destroy 'Evil' and define and promote 'Good'. Much of it is just a new form of Missionary work dressed up as something else, and IMO it's spread has resulted in continued and seemingly random destructive and tragic cultural shock waves.

To me these are phenomenon of a concept like Social Physics. 

Another part is whether it's possible for the assistant to follow instructions while also helping me reach my personal growth goals. I feel like there's some wiggle room there. 

 

I agree, although I can also see it functioning like court-appointed counseling or drug/alcohol treatment like we have in the US. If the user doesn't have the motivation to change their thinking/behavior, they just revert back to their 'old bad behavior' when their time is up, whether it's with the courts or with something like a court appointed AI rehabilitation assistant. 

Unless you make it also like an ankle monitor so that the AI assistant follows them everywhere at all times, but without the appropriate programming and complicated information architecture and security, that would cause all kinds of privacy concerns. It would also depend heavily on societies cultural development up to that point. 

As an example of what I mean, if these AI had been developed and sold/court ordered in the 80's when the War on Drugs was really kicking off and society thought and acted like all drug use was bad, then society would be much different today as a result. Ideas and institutions like Mass Incarceration and heavy sentences for drug offences might still be very popular, and the War on Drugs might still be accelerating for the foreseeable future instead of winding down as it seems to be. Ideas of what's Good/Bad, acceptable/not acceptable, healthy/not healthy, criminal/not criminal seemingly change with the wind as time goes by, and I honestly don't see that fact changing without several huge interventions.

That's why I argue for a concept of Social Physics, which cultivates the similarities between the "Hard" sciences and the 'Soft" sciences to form a rational understanding and underpinning of Social Dynamics as the result of the physical forces of science, but with a Cohumane spin: Cohumane being thought and action devoted to human endeavors seeking equality with all of nature, including alien life if/when it's discovered and AI if/when it achieves sentience/intelligence. I use this concept of social physics to try and think about the world in a more rational way.

The pattern of dominating technologically inferior cultures, eliminating and co-opting their cultures and making them slaves, until they either rise up and revolt, or their equality becomes apparent after their culture has been disintegrated, forces dominant society to spend huge amounts of energy to completely reshape society in often poor, ineffective and conflicting attempts at restorative justice. 

For instance, both the African population boom and Aids epidemic in the 80's and 90's which resulted in so many deaths, so much Famine and suffering, is a situation which continues 30-40 years later, to consume huge amounts of resources from the US. We are still struggling to meet the needs of our own people, and the interests of those at the bottom in US society often come into conflict with the needs of people around the world. It seems hypocritical to be fighting these issues around the world and espousing our ideals of fairness and equality to others, when we are dealing with them so ineffectively at home still.

For instance, I spent 18 months living in homeless shelters in the US, and what I saw reminded me very strongly of what I imagine a FEMA camp and a prison system would look like if they worked together in the heart of a mid-sized city. I'm still struggling in the shelter system over 2 and a half years later, but no one is really attempting to help me out except for the overworked, underfunded, old and decrepit public services system. 

I've literally thought at times about how to get NATO to come and intervene in the us because what I've experienced and witnessed at times seem to me to be clear violations of human rights. But, the belief is, the US is a 1st world county, and we don't have epidemics of hunger, poverty and disease; drug lords running the streets, or insurgents attempting to breach the capital and change the results of a democratic elections or things of that nature. It seems the commons sense is "That stuff happens in 3rd world countries, not the US." Maybe not now, at least with the Insurgency and the pandemic. 

Back to Africa though and their concerns, the fact that so many christian missionaries from the Western world were traveling all over the world and proselytizing, meant that when food, technology, and medicine arrived with these missionaries, it seems likely their religious beliefs about sex were also transmitted, so that condom use and other safe sex practices were not passed along, resulting in a population boom and the Aids epidemic. I've no data to back that up, but my research skills aren't that good. I'd be interested if anyone knows of any data and research dealing with this idea. After all this writing and editing and rewriting though, I'm too tired to even Google it. 

These things though could have been avoided completely, at best - or - at least the population boom and growth/spread of Aids could have been slowed significantly if the integration of the technology, food and medicine had also included counteracting safe sex practices as well. 

The US accounts for about 4.25% of the world population. As the 'leader of the free world' we don't have problems of overpopulation like some other countries,but we have other problems that we continue to struggle with, and we continue to pass those problems around the world along with the the rest of our culture.

IMO examples like this continue to happen because of the blind spots of Conservative Christian values which have dominated so much of American legislation. The interaction of these conservative efforst and the reflexive overcorrection's by Liberal activists, polictician's and citizens, result in  the continued culture wars and the reverberations of these battles are magnified and accelerated by the Internet and contemporary technology. Psycho-socio-economic faults in the Cultural Tectonic Plate Formations which constitute the foundations of the now shared cultural sub-conscious, continue to cause politcal Tsunamis and cultural earthquakes which shake the very foundations of our contemporary societies.

A Technology/Science of Ethics, not morals, as I think of them, would include ideas somewhat along the lines you are proposing. They would explicitly seek to figure for mistakes of the past, and attempt to correct for possibilities in the future, hopefully avoiding potential catastrophies and rebalancing the current social systems by analyzing and including rational consideration of blind spots in the dominant cultures conceptions of what is right/wrong, good/evil, and criminal/non-criminal. I can't do it with programming or programming logic, but I'm pretty sure this is what most rationalists are concerned with.

What if the world really isn't binary? What if humans need to do bad things sometimes for good to come out of it, or if sometimes good intentions result in Global catastrophe? I believe it's possible that the heart of the matter is that binary thinking is a false dichotomy, and while our technology has become super advanced based of these principles, Cultural Lag has kept the majority of the worlds growing population from benefiting in a Cohumane way, by unintentionally exacerbating problems like over population.

I like the idea of the AI assistant reminding the user of their previous conversation and stated goals of self betterment, and I like to think that this is the type of future humanity might have. I would love to consider this idea in a more positive way, but my current circumstances make me wonder how we would implement it all around the world at the same time? 

I often feel as an American, I am constantly being asked to consider the problems of the rest of the world as being more important then my own though, and I wonder how the interests of the 4.25% of the world population that the US represents would fare in 'equal' or 'fair' relations with the other 95.75% or the world if an AI like the one you describe were developed tomorrow. 

Some of us are already at the bottom of the socio-economic ladder in the US, and at this elevation, it looks a lot like what I've seen of extreme poverty, deprivation, physical/mental health, disease, hunger, violence, and repression around the rest of the world from tv. In a world like the one we currently inhabit, I don't believe the development of an AI system like this would do anything but increase the ever widening gap between the haves and the have-nots.

I really do think the idea of semi-autonomous Ai using 'fairness' protocols to engage in cooperative beuracratic bargaining to encourage the development of 'fairness' (whatever that would be) is an interesting idea though. That's a usage I'd never contemplated until now.