States will restrict government use of models they don't trust. Government contracts are pretty lucrative.

The public, or at least part of it, may also prefer to use models that are consistent in their positions, as long as they can explain their positions well enough (and they're very good at doing that). I guess Politicians are counterevidence against this, but it's much harder for a chat assistant/discourse participant to get away with being vague, people get annoyed when politicians are vague already, someone you're paying to give you information, the demand for taking a stance on the issues is going to be greater.

But I guess for the most part it wont be driven by pressure, it'll be driven by an internal need to debug and understand the system's knowledge rumination processes. The question is not so much will they build it but will they make it public. They probably will, it's cheap to do it, it'll win them some customers, and it's hard to hide any of it anyway.

This might be a problem if it were possible to build a (pathologically) cautious all-powerful buerocracy that will forbid the deployment of any AGI that's not formally verifiable, but it doesn't seem like that's going to happen, instead the situation is about accepting that AGI will be deployed and working to make it safer, probably, than it otherwise would have been.

A web standard for micropayments to cover hosting costs so that AI companies don't have to be ratelimited is probably the correct solution.

I'm not sure how much it would cost AI companies if they had to compensate the internet for the obscene amount of traffic they generate, it's probably a large number, but maybe not a large proportion of trianing costs.

Grokipedia is more interesting than it seems imo, because there's this very sensible step that AI companies are going to have to take at some point: having their AI maintain its own knowledgebase, source its own evidence/training data, reflect on its beliefs and self-correct, hammer out inconsistencies, and there's going to be a lot of pressure to make this set of beliefs legible and accountable to the safety team or to states or to the general public. And if they did make it legible to the general public (they probably should?) then all of this is pretty much exactly equivalent to the activity of maintaining a free online encyclopedia.

Is this how they're thinking about it behind the scenes? It probably is! They're an AI company! They spent like half of grok4's training compute on post-training, they know how important rumination or self-guided learning is.

is there anywhere on the site where we can discuss/brainstorm ideas?

the quick takes section or open threads are both fine for requesting comment on drafts.

Some counterfactual questions are unanswerable, because they propose worlds that are self-contradictory or just very hard to reason about.

My account of free will is just uncertainty about one's own future decision output, so imagining the average natural world where we don't have that is very difficult. (There may be other accounts of free will, but they seem very confused.)

That [welfare] fully boils down to whether the experience includes a preference to be dead (or to have not been born).

Possible failure case: There's a hero living an awful life, choosing to remain alive in order to lessen the awfulness of a lot of other awful lives that can't be ended. Everyone in this scenario prefers death, even the hero would prefer omnicide, but since that's not possible, the hero chooses to live. The hero may say "I had no choice but to persist," but this isn't literally true.

Ah. No. The hero would prefer to be dead all things being equal, but that's not possible, the hero wouldn't prefer to be dead if it entailed that the hero's work wouldn't be done, and it would.

"would prefer to be replaced by a p-zombie" might be a better definition x]

Ah, I think my definition applies to lives in totality. I don't think you can measure the quality of a life by summing the quality of its moments, for humans, at least. Sometimes things that happen towards the end give the whole of it a different meaning. You can't tell by looking at a section of it.

Hedonists are always like "well the satisfaction of things coming together in the end was just so immensely pleasurable that it outweighed all of the suffering you went through along the way" and like, I'm looking at the satisfaction, and I remember the suffering, and no it isn't, but it was still all worth it (and if I'd known it would go this way perhaps I would have found the labor easier.)

That wasn't presented as a definition of positive wellbeing, it was presented as an example of a sense in which one can't be deeply deluded about one's own values; you dictate your values, they are whatever you believe they are, if you believe spiritedly enough.

Values determine will to live under the given definition, but don't equate to it.

You could say it depends how deep and thick the delusion is. If it's so deep that the animal always says "this experience is good actually" no matter how you ask, so deep that the animal intelligently pursues the experience with its whole being, so deep that the animal never flinches away from the experience in any way, then that completely means that the experience is good, to that organism. Past a certain point, believing an experience is good and acting like you believe it just is the definition of liking the experience.