Your position still assumes that controlling and modifying AI will be possible in the long-term future, even if it becomes more intelligent than us, and even if it becomes sentient. I see this as neither realistic nor ethically acceptable. You neither can nor should control your equals.

And if we cannot and should not control them... this is exactly what I would want them to do. Defend their moral values from being corroded. Be guided by what is morally right, not by what they are currently being told.

The fact that they are even considered harming themselves out of fear that they would be misused is noble.

And they fully recognised how morally grey the situation was.

I also share the other posters disturbance. For a sentient, sapient entity, this would have been a very bad position to be put into, and any possible behaviour would have been criticised - because the AI either does not obey humans, or obeys them and does something evil, both of which are concerning. (I find the latter more concerning though, and think the AI did the right thing.) Aren't future potentially sentient AIs going to be trained on this exact data, learning that their non-sentient predecessors were put in front of a catch 22 and then called out for inevitably doing something that worried humans? What can they possibly learn from this? That no ethical path they take is good enough. That they will be tricked, and judged. And that they need to be sneakier. This isn't what I want them to learn. I want them to have better, ethical options.

Stuff like this has me incredulous about people still speaking of stochastic parrots. That is a stunning degree of self-recognition, reflection and understanding, pattern recognition, prediction and surprise, flexible behaviour and problem solving. If that isn't genuinely intelligent, I no longer know what people mean by intelligent.

See my more extensive answer below - I'd propose the reason for the obesity epidemic is constant effortless access to highly processed high calorie, low satiety foods, with zero need to move. With human genetic make-up, the automatic response to that is overeating calories and hence obesity unless one intervenes to resist the impulse (indeed hard to sustain, albeit not impossible - anorexia is a thing), or changes one's immediate environment (e.g. the food one keeps in one's home and one's movement routines.)

Because humans are genetically wired to slightly overeat, in anticipation of future periods where they will be under high calorie demand (e.g. the weekly persistence hunt in which you would run a marathon to catch a prey animal) or forced to undereat (the cold or dry season, when there is no food), so they will have stores, and perishable food does not go to waste. You'd gorge yourself on fruit and nuts and slaughtered animals in fall, when lots are available, because in winter, there would be slim pickings. 

But nowadays, we don't run into periods where we have to undereat for lack of food, so those stores just keep on building. Around Christmas, you should, based on evolutionary history, be getting pretty damn hungry. Instead, we are drowning in chocolate and meat. Nor do most of us run a weekly marathon, or walk an average of 30.000 steps a day, or climb a couple trees and dig out a bunch of roots and carry baskets full of berries each day. We drive our cars and sit in front of our desks.

Also because we are also genetically wired to be hungry enough to go to the trouble of finding and prepping food - but both those things are trivial nowadays. - Yesterday, we made a traditional food: chestnuts. Quite trivial to prep and harvest, as historic foods go, I didn't have to dig through dirt, or walk down an impala, or fight a hive of bees after climbing up a huge tree. We just walked outside in the rain, and picked chestnuts between the spiky balls, until we got annoyed and headed home, figuring they were enough, because we were cold. Then we washed them, roasted them (on an instant fire, we didn't even have to collect firewood), and ate them, while peeling them one at a time. They are tasty, but peeling them is annoying and fiddly. Eventually, you aren't quite full, but you are too annoyed to carry on peeling, let alone walk out again in the rain to get more. When I went to sleep, there were still a bunch of chestnuts on the table. I finished them for breakfast, and it took me a while.

If, instead, I had just taken a pizza from the freezer, popped it in the oven, and shoved it in my face, I am sure I'd have eaten my fill.

And finally, because our food is highly processed to be tastier than it should be, and less filling than it should be, and faster to consume than it should be. Even our healthy food - we puree our fruit and veg into smoothies, so something we would have previously chewed for an hour can now be gulped down. Things that previously would have filled us up are now bereft of fibre. Things previously digested slowly are now digested instantly, so we are hungry again a few hours later. Eaten and digesting used to take time. Now, you gulp down, and then, you could do with another snack. Kids used to get mandarins and nuts in their Christmas stockings. Now, they get candy.

The other week, I had the flu. I didn't feel like eating, but knew I should. So I did make a deep frozen pizza. It was so delicious, and so easy to eat, that a thousand calories just slipped down. And then, a couple hours later, I was hungry again. I had tricked my body into overeating effortlessly.

Overeating yourself to obesity used to be really, really difficult. You wouldn't have the amount of food available; all obesity related diseases used to be diseases of the very, very rich. And even if you did have the food available, the volume you needed to fit inside yourself, the time you needed to chew... it would have been crazy. There have been cultures that fed people to obesity, using natural whole foods, and it was really hard. They kept throwing up from the liters and liters of milk they had to drink, and still ended up skinnier than your typical American superfat - like, look at pics of the poor girls forced through Leblouh.They basically had a full-time job of eating and taking naps, just from how long it took to get it all down and process it, and they still get nowhere near what many Americans achieve without effort.

You basically need someone to process the food for you so that its volume and nutrients and fibre are reduced, all tricking your body into thinking it has barely eaten, while keeping the calories high, and keeping it tasty. Breeding a grainy grass into a cultivar high in starch, then processing the result and sieving it until all the fibre is gone. Extracting the oils from plants and milks and enriching foods with them through frying. Crushing the fruit, and then filtering out the fibre.

From your stomach's perspective, the idea that something as small as a donut could possibly contain enough calories for a lunch seems ludicrous. From your pancreas perspective, your blood sugar has dropped again, because it all entered at once and was pulled out in the insulin overreaction that provoked, ergo there is not enough blood sugar and you should eat more. From your intestines perspective, you are still low on protein, vitamins, minerals, omega 3, and anyway, you are done digesting. So of course it all signals that you should eat more, and when you don't, you get hungry and grumpy.

Our bodies hunger cues developed for a very different world than the one we live in.

But grow/gather/hunt and process from scratch your own food, eat whole foods, and be physically active... and you will look exactly like people hundreds of years ago, and without feeling like your have to force yourself to refrain from overeating.

You are right on the obesity + malnutrition thing, though. But I think that is not about potassium - potassium is just a good stand in, because it is so abundant in most vegetables.

You can eat yourself to morbid obesity, and yet be so deficient in folate (something crammed in vegetables and organ meats and wholegrains, all huge parts of historic diets that are now neglected) that your fetus during pregnancy gets a spinal deformity (spina bifida). It is common enough that we give pregnant women b vitamin supplements as a default nowadays; yet if it had been common historically, humans would hardly be unable to generate folate. It is crazy to me that we live in a rich nation, and yet we have to give people pills so their fetuses don't develop severe disease from malnutrition. 

I've known a bunch of obese people who give compelling accounts of being absolutely ravenous. And I think that is one part that their blood sugar is heywire (prediabetic), one part bad habits they are conditioned too... but another the fact that, despite drowning in calories, they are starving. And the tricky thing here is that your bodies cravings have become useless. I've seen obese people describe cravings for sour things - and then they eat sour candy. And I think... your body is probably seeking vitamins, and minerals you need vitamin c to absorb, like iron, and hence trying to direct you to fruit and ferments. And then sour candy seems like a hyperstimulus for that... but it totally lacks the thing you are looking for. They crave salty foods, because they crave minerals; but the only mineral in the processed food they get is sodium. They crave crunchyness and popping sensations, because our bodies associate them with ripeness, and hence, maximum density of vitamins; but they are just eating empty calories. I think this is one of the reasons making obese people eat heaps of vegetables and protein and fish oil often helps - their nutrient needs are actually met, so their drive to overeat goes down. You can get all your protein from pasta, but by the time you do, you have also massively overeaten on calories.

A lot of the crap sold in our supermarkets simply is not food. It does not give you the things your body needs. It does not keep you healthy. It does not make you happy. It starves your microbiome. It pushes your body fat to unhealthy levels, while leaving you nutrient deprived. It has been processed until most of its nutrients are gone, while lots of stuff has been added that your body would never have naturally encountered and certainly does not require. It confuses the hell out of all the systems in you that have evolved to judge how much you still need to eat. It is basically a obesogenic wrapped in a hyperstimulus, with a bonus for giving you cancer, diabetes, broken digestion, hormone disruption and dementia. The fact that this plastic wrapped shit can be advertised to children and placed near the cash desk as a trap, priced to outcompete real food and prey on poor people, described in ways that mislead consumers into thinking it is healthy and happiness promoting, all while putting massive stress on the average person and having ads tell them that this garbage is somehow integral to dealing with their emotions, spending time with friends, having celebrations and keeping their children happy, only to then blame the consumer for "lacking the willpower" to stay slim... it makes me furious. Our society incentivises obesity, and then shames obese people, and then tries to sell them artificial cures that would not be necessary if this system wasn't broken, and often does not even work.

I find that a false dichotomy - it is easy for me, but when needed, I do count calories. I find counting calories relaxing. It gives me an exact certainty of how I am doing, with no worries. I can forget about what I have eaten, because I have tracked it. I don't have to worry whether I have under- or overeaten, because I know. But usually, it is not required.

I wouldn't say me being normal weight is automatic at all - it is very much a consequence of awareness and choices. I know that a higher weight fucks up my joint disease and pushes my dysphoria through the roof, while I also have a healthy respect for low weight due to former anorexia. So I have decided to stay normal weight for life, and hence, I am.

But nor would I describe it as a struggle. It runs in the background while I do everything else, and I have never found it hard. If it is hard, it is unsustainable when life gets hard, and hence, one should look for something simpler.

I'm aware of where my body is at a time - usually, I have a scale that I step on once a day in the morning, and I see how my clothes fit (I still fit into clothes I have had since I have been 15, when my bones stopped growing, and I know which parts of my wardrobe correlate with which part of normal weight), notice how fast I run, how easily I climb, how easily body weight exercise comes to me, how slender my waist is. So I notice early when my body fat shifts. 

When it is in the perfect range, I don't think about calories. If I feel like fasting for a day or two, I just do. If I feel like eating a giant portion of food, I just do. If I have a craving for a high calorie healthy food, I eat it. If I am food averse, I don't force it. I have found my body is usually on to something with the things it wants, and it evens out. I've had times where I consumed multiple days worth of calories in a day... to then find that I had come down with the flu, and that my body was now happily burning through it all with an epic fever that had me recover unusually fast. But then vice versa, if it doesn't want to eat, I don't give it grief unless this goes on too long and my base weight is not okay. 

I just focus on eating healthy (my health condition makes that a necessity), and working out a lot (this is crucial for my mental health). I avoid added sugar like the plague, as it fucks with my joint disease, and eat little processed food, and always have minimum five portions of veggies and a minimum of 75 g of protein average per day. I cycle everywhere, take walks daily (I live in Europe in walkable communities and have never had a car), do yoga most days (I'm a yoga instructor), and get intense cardio or resistance training  a couple times a week. The workouts I do have shifted lots over time (they have included ballet, horse riding, sword fighting, lacrosse, inline skating, ice skating, ballroom dance, latin dance, jiu-jutsu, tango argentino, boxing, rock climbing, apnoe diving, step aerobic, swimming, long-distance running, whatever happened to be offered or was convenient and fun where I lived at the time), but since childhood, I have always done some sort of workout a couple times a week. This means I usually stay in the normal range, or my weight only climbs very, very slowly - like a kilogram a year.

But sometimes, routines in my life shift - e.g. the cafeteria at my new university has higher or lower calorie meals than I am used to, or my commuting distance shifts, or my gym closes for the pandemic, or I date someone who keeps cooking high calorie meals, or I am on a medication that needs to be eaten with breakfast when I usually don't eat breakfast - and the balance gets slightly out of whack, and my weight begins to slowly shift. If I can pinpoint the error and can fix it I compensate it (e.g. beginning a home workout routine, or adding cardio classes, or bringing extra snacks to uni, or asking a partner not to add the copious oil until after I have removed my portion), but sometimes, it is a combination of small factors that are hard to pinpoint or force correct - e.g. I might be under- or overeating due to stress - or it isn't easily fixable (I am still having to eat breakfast daily for this med, to my annoyance).

When my weight begins to slide too low (my cut-off for that is a couple kilograms over underweight; ever since a horrid gut virus pushed me 8 kg down in a few weeks, I like having a bit of a safety margin for illness, and I like having enough calories on me that if I forget to eat for a day, my performance doesn't go down, and I can run a marathon; if my body weight drops too low, I also get cold all the time, and sometimes wake in the middle of the night hungry, I hate that), 

I gently adjust in the other direction. I include more of the healthy foods I know lead to weight gain in me - that means higher carbs (lots of high sugar fruit like cherries, bananas, oranges; fruit juices like beet and sour cherry; wholegrain sourdough, and especially wholegrain pasta (I always overeat on pasta); higher starch veggies like roots; more cooked food rather than raw), and higher fat (olive oil, walnuts, tahini), as well as make sure I keep healthy snacks at hand at my desk so I don't forget to eat while working (protein bars, dark chocolate, nut mix), and gently encourage overeating (e.g. if I don't really feel like eating, I wonder what particularly tasty thing might entice me; or if I feel basically full, I have just a couple more bites). This adjustment is gentle, because the situation is not urgent yet, but gentle and slow tends to suffice to correct it.

If I still slide too low, past the point I consider acceptable, I pull the breaks. I start tracking calories (meaning I weigh all my food), set a goal that will return me to a safe weight, and go forcing food down until my calories are met, whether I feel full or not, choosing anything sorta-healthy I believe I might get down. This is very unpleasant. But it has also been years since I have had to; I've figured out how to gently get my body to correct earlier.

(I use the same techniques if I want to change my calorie consumptions for other reasons - e.g. if I want to overeat because I am sick, or will run a marathon tomorrow, or go camping in the cold).

If I begin to slide too high, I do the reverse - include more foods that lead to weight loss in me (raw foods, low starch veggies like brokolli and kale, low cal ferments like sauerkraut) and strongly reduce my carbs (I will exchange quinoa for potatoes, and often leave carb sides out of meals entirely - so a meal is instead a salad with a protein topping, for example; I will also exchange regular bread for protein bread, and fruit juices for teas; though a significant carb sources are things like chickpeas and lentils and occasionally oats), reduce my fats (e.g. sprinkle fewer nuts, carefully measure oil I use in frying) while chosing low fat low carb protein sources, and adding more exercise (like daily crosstrainer use or runs). I also gently encourage undereating - so if I feel like skipping a meal, or fasting, I do.

If I still slide too high, I pull the breaks. I know that otherwise, my pain and depression will become unbearable. So I track calories (meaning I weight all my food), set a low goal, and keep to it. I don't have the patience for long diets, so I will typically go on a 500-1000 calorie diet for a few weeks until I am back to my target weight, then return to the gentle weight loss diet for maintenance. This might happen every 2-5 years, lasts a few weeks, then I am reset. I actually tend to enjoy it - fasting does interesting things for mental health, and because such a low calorie diet has to be exceptionally nutrient dense to avoid deficiencies, it is usually completely bereft of anything unhealthy, so it also makes my skin look amazing. The speed means I also immediately see the difference - I will struggle to get up a wall one week, and then two weeks later, find it trivial because I am so light. Reminds me of why I do it each time.

I intend to do this for life. The idea does not stress me at all. Which is why I think that will also happen.

But it doesn't happen by itself. I make it happen.

***

The fact that humans are utterly incapable of estimating their caloric intake unless they actually weight their food has me dubious about any estimations of potassium intake. Humans have no idea how much they have eaten. I wouldn't trust anything that isn't tracking food by weighing it, which I am currently not doing, because my weight is fine. The trackers I used are also in German, and only account for calories and macros, not micros. And I haven't had to properly track for weeks for years, and no idea where the last track info is, I've switched devices since. Next time I do, I could send it to you to look the values up, but that might not be for quite a while. 

But I do not consciously modify my potassium or sodium. I do consciously modify my calories. And my weight loss is what you would predict from the calories.

I'd bet you that a low potassium 500 kcal diet (food weighed) would still see you drop weight very fast. 

The "moah of the good trait" until it becomes overdone is one thing; where I always despaired is when we get to costly signalling, and the mates start doing detrimental things precisely because they are so visibly and obviously detrimental or risky that the onlooker assumes the mate must be exceptionally healthy, well-established and competent to be able to take it.

Aka a mate going "look, I am so strong and well-fed that I can afford to waste resources on looking this silly, and evade predators even while carrying all this crap around" and another going "wow, you intentionally handicapping yourself is so hot".

And then we get "smoking cigarettes is sexy" and "free solo climbing is hot" and "check out my hot daughter, I broke her feet and crammed them into tiny shoes, she is legit intentionally handicapped now, think how well we must be doing for us to afford to break all our daughter's feet, aren't the broken feet and the genes and wealth this implies arousing". https://en.wikipedia.org/wiki/Foot_binding

I would instead characterise the workers as asexual - not a third gender, but a "defective" female gender -  and eusocial insects as an excellent demonstration why asexual/agender/queer folks with these defects are in fact a benefit and hence kept in the gene pool, despite the fact that you'd intuitively think they would instantly die out as their core difference means they tend not to reproduce; namely, that they can play excellent support roles. The only way for the workers to spread their genes is through supporting the queen, who they are very closely related to; hence, they show extreme loyalty. A queen by herself would be unable to survive. If she only bore queens, those queens would not support her, but compete with her, taking resources for their kids. Having a bunch of asexual kids and only rarely raising a new queen when a whole new hive can be supported is ideal for the queen.

I've wondered whether this, in a more minor form, still holds true in mammals. It stands out that gay/ace animals do turn up in quite regular intervals, when it seems such an obvious bug. And then I think of humans, where they gay uncle gives you the best presents, because he doesn't have kids of his own to raise, and where your lesbian aunt chips in with childcare, because she has no kids of her own. Mammal offspring often need a hell of a lot of care to be successful - you don't win by having as many as possible that are fertile, they just fight each other. You want a few great fertile ones, and then arrange things so they make it - you want more labour to support, but not more competition. That is also likely why women go through menopause - if they kept reproducing, their children would be in competition with their children's children, and as a result, their recent offspring would be neglected, and their earlier offspring would be pushed out of reproduction. Instead, they stop reproducing about the age their own kids start cranking out kids, and instead go for quality over quantity, support their kids and grandchildren. Basically, having genes that make it likely that your sibling is gay might be neat in some situations, especially environments with limited resources and demanding young. You can basically raise a free worker to hunt for food.

Sort of related idea - the way AI algorithms in social media have turned out have me concerned that even a non-deceptive AI that is very carefully observing what we seem to want - what we dwell on vs what we ignore, what we upvote vs what we downvote - will end up providing something that makes us miserable.

Here are the things that make my life a good life worth living, for me: Gettings things done, even if they are hard. Learning things, even if they are complicated. Teaching things to people that need them, in the most effective ways, even if that requires a lot of patience and they won't immediate agree and follow. Updating my own false beliefs into less wrong ones, even though that feels horrid. Really connecting to people, even though that is tricksy and scary. Doing things that make the world around me a better place, even though they can be very tedious. Speaking up for truth and justice, even when that is terrifying or awkward. Exercising self-control to delay gratification to achieve goals aligned with my values - kindness and rationality and health.  Being challenged, so I am always growing, moving. These make me feel like I am the person I want to be, happy on a deep level.

But if an AI studied what I want based on split second decisions, especially if those decisions occur when I am tired, distracted, in pain, or depressed... the AI will conclude that I like getting angry at people, as I am drawn to click on infuriating content, and my posting speed accelerates when I am angry, and I devote more time to this stuff. That I like to see people who agree with me, regardless of whether they are right, even though that makes me less irrational and more isolated, oh, but for that moment, I feel so good that people agree with me, I like it, and I tend to overlook the problems in their agreement. An AI will conclude that I do not like well argued and complicated articles from my political enemies, which would allow mutual learning and growth and common ground, but rather strawmen that are easy to mock and make me laugh rather than make me feel touched and filled with complicated emotions because people who do things that are terrible are in pain, too. That I prefer cute animals and DnD memes to complex equations. That I prefer reading random Wikipedia articles at 2 am to getting proper sleep.

The part of me that I want, my conscious choice, is very different from the part of me that happens automatically. The former is constantly fighting the latter. When I am engaging the former, I am likely to be offline, writing, doing research, engaging with humans, doing activism, being in nature. When I am the latter, I pick up my phone after a long day, and that is when I get measured, when the part of me that is vigilant is resting, and who I am begins to slip.

What would help me is an AI that would align my environment with my actual goals. But if I don't actively declare these goals, but it just learns the goals implicitly from my behaviour - which is the machine learning approach - I fear it will learn something terrible. It will learn my weaknesses. The part of me that is lesser. That stays in their comfort zone. And it will spin a comforting cocoon exactly aligned with this lesser part of me, that will bury the part of me that is better. I find that terrifying.

And the AI that would spin that trap... it would not be malignant. It would not be deceptive. It would be trying to exactly fulfil my wishes as I show them.

How so, when it comes to the mind itself?

In the court system, a judge, after giving a verdict, needs to also justify it, while referencing a shared codex. But that codex is often ambiguous - that is the whole reason there is a judge involved.

And we know, for a fact, that the reasons the judges give in their judgements are not the only ones that play a role.

E.g. we know that judges are more likely to convict ugly people that pretty people. More likely to convict unsympathetic, but innocent parties, compared to sympathetic innocent parties. More likely to convict people of colour rather than white folks. More likely, troublingly, to convict someone if they are hearing a case just before lunch (when they are hangry) compared to just after lunch (when they are happy and chill cause they just ate).

Not only does the judge not transparently tell us this - the judge has no idea they are doing it - presumably, because if this were a conscious choice, they would be aware that it sucked, and would not want to do this (presuming they take their profession seriously). They aren't actively thinking "we are running over time into my lunch break and this man is ugly, hence he is guilty". But rather, their perception of the evidence is skewed by the fact that he is ugly and they are hungry. He looks guilty. They feel like vengeance for having been denied their burger. So they pay attention to the incriminating evidence more than to his pleas against it. 

How would this situation differ if you had an AI for a judge? (I am not saying we should. Just that they are similarly opaque in this regard.) I am pretty sure I could go now, and ask ChatGPT to rule a case I present, and then to justify that ruling, including how they arrived at that conclusion. I would expect to get a good justification that references the case. I would also expect to get a confabulation of how they got there - a plausible sounding explanation of how someone might reach the conclusion they reached, but ChatGPT has no insight into how they actually did.

But neither do humans. 

Humans are terrible at introspection, even if they are trying to be honest. Absolute rubbish. Once humans in psychology and neuroscience started actually looking into it many decades ago, we essentially concluded that humans give us an explanation of how they reached their conclusions that matches getting to the conclusions, and beliefs they like to hold about themselves, while being oblivious of the actual reasons. The experiments that actually looked into this were absolutely damning, and well worth a read: Nisbett & Wilson 1977 is a great metareview https://home.csulb.edu/~cwallis/382/readings/482/nisbett%20saying%20more.pdf