Dagon

LESSWRONG
LW

Dagon — LessWrong

While I do fully support and experience differential individual weighting in my utility, I'm not sure I understand what would justify the idea of "cosmic utility". I don't believe there is any shared universal (or cross-universal) experience that really corresponds to a valuation or valence. Utility/preference is individual, all the way down (and all the way up).

I think there IS a different asymmetry that can make status (and most interactions that appear zero-sum in resources) not actually zero-sum for the participants: mapping of shared/objective world-state to individual perceived status-value. It's possible that if participants are thinking of slightly different dimension of what increases or reduces their status, that many changes can increase A's (perceived) status more than it decreases B's. I think this is the standard "private utility function" problem very often mentioned in decision theory. You don't focus on this in your post, but I think it's the stronger model.

Dagon3d

Sorry for lack of clarity in my retraction - I'm pretty sure I'm right about my preferences - I am usually looking for recency. That belief hasn't changed.

I have retracted my annoyance - I should not be, and no longer am bothered by your use of "you" when it's really "most users" or "our target user". I hope it was useful for me to give my common use case, but I don't intend to demand anything - this topic is peripheral enough to my value from LessWrong that I wanted to reduce, rather than increase, any load I'm imposing.

Dagon3d

In fact, you're absolutely correct - I retract that part of my response. I also recognize that I use this feature (looking at someone's profile to see their recent posts/comments) less often that others might use it for other purposes (looking at their best/top contributions).

Too many user-options is worse than slight misses on some defaults. I'm grateful for LW, even when it's optimized for people who are not exactly me. Carry on experimenting, and see how people react and change their usage.

Dagon3d

I am pretty confident you actually pretty reliably care more about what the highest karma/self-curated posts from a user are, than what their most recent posts are (which is what we've always shown first so far)

Aside from bristling a bit at being told that what I care about is wrong, I actually do care about recency more often than historical top-ranked for this view. Of course, sometimes I'm looking for "best of", but more often I'm checking whether the user is in a snit or otherwise commenting recently in a pattern I should understand/accommodate before I join one of their threads.

Replying toIs a constitution a "noble lie"?

Dagon4d

Is a constitution a "noble lie"?

To a great extent, the underlying "noble lie" is that there is any such thing as objective moral truth. There is no measurement of "should", it's just about what equilibria seem to work, which is based on most people accepting it without questioning too hard.

Replying toShould we consider Meta to be a criminal enterprise?

Dagon5d

Should we consider Meta to be a criminal enterprise?

If you have some mafia element that runs a casino

I mean, you've embedded the answer in the question, by definition of "mafia", I think.

The pizza joint does not provide a way to target specific demographics who are likely vulnerable to scam attempts and run complex machine learning algorithms that optimize the effectiveness of the scam and auction of the marks to different scammers.

They might, if they're in San Francisco and signed up for the SaaS version of business optimization.

Dagon5d

I'm sympathetic to the model - in many cases it seems that there is a generalizable trait of "toughness" (a few decades ago they called it "grit" or "determination", now it rhymes with "agentic"). It's tempting to simplify things to that level.

But I'm also skeptical of my own desire to believe that, and I don't actually think it's true often enough to count on it. When I press myself on edge cases or most specific data->prediction proposals, it loses a lot of appeal.

There are clearly some people who fare better than others across many domains. Exactly which traits cause this, and how ingrained and unchanging those traits are, remains quite difficult to pin down. Personally, I think it's 50% luck, 50% genes, 50% early environment, and 50% current environment. Yes, success is overdetermined :)

Replying toShould we consider Meta to be a criminal enterprise?

DagonFeb 10, 2026

Should we consider Meta to be a criminal enterprise?

No. "criminal enterprise" is not a useful designator for this. Their primary business is not criminal, and even as an ancillary business, it's so deeply commingled with legit advertising that it's very hard to prosecute them as even negligently supporting those crimes.

Also, your local pizza joint likely takes money that was criminally acquired. They are also not a good target to designate as a criminal enterprise.

If you want to be a bit more nuanced and debate whether we should devise and impose some "know your customer" type rules on advertising brokers (or pizza parlours), that could be interesting.

-5

Replying toCoping with Deconversion

Dagon5d

Coping with Deconversion

"I'd rather be sad than wrong."

What's helped me is to realize that "sad" and "wrong" are different dimensions, and don't need to be correlated at all.

I grew up in a less-community-intensive church than LDS, and my intellectual beliefs have been purely atheist since my mid-teens. My professed beliefs among some groups of family and community is more agnostic, and I'm very comfortable with others believing things that seem unlikely to me, as long as we can still cooperate in having fun and improving the world (which means I'm not part of groups that demand explicit declarations I don't believe).

Happiness for most humans does require community and love from other humans.... (read more)

Replying toStone Age Billionaire Can't Words Good

Dagon6d

Stone Age Billionaire Can't Words Good

do you think that you've identified a set of characteristics that, throughout history, will never lead to the wrong conclusion?

Not at all. There is no such set of characteristics. Wrong conclusions are inevitable and commonplace. Godel's Theorems apply to all formalisms.

In the current world, the harm of unsanctioned killing being commonly accepted (and cheered) is generally a LOT higher than the harm of statistically-evil people continuing to live. So, yes, a heuristic argument: this is a loss of civilization and order, even if it might have been justifiable on some dimensions.

Moral realism - basic Q

Dagon

7mo

I'm not a professional philosopher. I've done a lot of reading and thinking about decision theory, economic preferences and behaviors, etc. I think I've absorbed most of LW from the beginning.

We often have posts exploring morality, and I find the premise that it's anything but a social consensus of human preferences to be ... incoherent. At least outside of the possibility of a deity or other super-physical causal structure.

For those who give significant weight to the possibility that morals are real, in the "acually have some effect on the universe" sense, can you give me a sketch of what kinds of impact it has? Even if we can't measure it currently, what are the units we'd use if we could? What IS it, if not "just" our beliefs and preferences about each others' behaviors?

Reminder to all: thought experiments are limited in what you can learn. Situations which are significantly out-of-domain for our evolved and trained experiences simply cannot be analyzed by our intuitions. You can sometimes test a model to see if it remains useful in novel/fictional situations, but you really can't trust the results.

For real decisions and behaviors, details matter. And thought experiments CANNOT provide the details, or they'd be just situations, not hypotheticals.

What epsilon do you subtract from "certainty" in your own probability estimates?

Dagon

Ok, nobody is actually a strict, or even particularly careful bayesean reasoner. Still, what probability do you reserve to "my model doesn't apply, everything I know is wrong"? If you SEE a coin flip come up heads (and examine the coin and perform whatever tests you like), what's your posterior probability that the coin actually exists and it wasn't a false memory or trick in some way?

Should LW suggest standard metaprompts?

Dagon

Based on low-quality articles that seem to be coming up with more regularity, and as mentioned in a few recent posts, AI-generated posts are likely to be a permanent feature of LW (and most online forums, I expect). I wonder if we should focus on harm reduction (or actual value creation, in some cases) rather than trying to disallow something that clearly people want to do.

I wonder how feasible it would be to have a LessWrong-specific workflow for using any or all of the major platforms to assist with (and not fully write) a LW question, a LW summary-of-research post, or a LW rationalist-exploration-of-a-question post (and/or others). This could simply be a... (read more)

If I make a comment, then the author deletes the post, is my comment lost as well? I'm pretty sure I don't actually need a record of things I've said, and that my comments aren't valuable enough to justify extra effort, but it kind of bugs me.

Has anyone written the scraper or script to save a copy of everything you write on LW, so it's available even if it gets deleted later? If it archived the post and comment thread you're responding to, it would be all right.

There's a large class of social/cultural behaviors that I think are fine and good when they're somewhat rare, and I find annoying and then harmful when it becomes very common.

This includes public demonstrations, minor mostly-victimless crimes, many kinds of drug use, tipping expectations, and I've just realized that April Fools is now in this category.

It's amazing when there are a few surprising well-crafted jokes published. It's annoying when it's common enough that it's hard to find the good ones, and even harder to find the non-joke communications.

</rant>

Comment throttling seems to catch people offguard somewhat often. I suspect this is because the criteria are imperfect, but also because it's a VERY sharp change from "unlimited" to "quite small".

Perhaps there should be a default throttle of 5 or 8 comments per day. And this throttle tightens with net downvotes. Maybe increase it with recent net upvotes, or maybe only by an explicit exception request with an explanation of why one wants to make more than that in a day.

edit: oh, better - do away with the daily granularity. Everyone has a leaky-bucket throttle, with say 4 comments max available, which replenish one every 2 hours. If your recent karma average is low, the replenishment slows down. If your recent karma is high, the replenishment speeds up. the bucket size could go up for posters with significant semi-recent upvotes.

I'm a bit uncomfortable with [edit: wrong word. all discussion is good, but I think there's a big modeling error being made.] a lot of the discussion about ethics/morals around here, in that it's often treated as a separate, incomparable domain to other motives/values/utility sources.

It may be, in humans and/or in AI, that there are fairly distinct modules for different domains, which somehow bid/negotiate for which of them influences a given decision by how much. This would make the "morals as distinct evaluations" approach reasonable, though the actual negotiation and power weights among modules seem more important to study.

But if something approaches VNM-rationality, it acts as if it has a unified consistent utility function. Which means an integrated set of values and motivations, not separate models of "what's the right action in this context for me".

So, either AI is irrational (like humans), or exploration of morality as a separate topic from status, pleasure, family, or other non-altruistic drives is not very applicable.

I'm torn between two models of recent technological change. In some sense, the internet from 1992 (hypertext is a cool organizing principle) to early '00s was a transformative societal event, and it arguably destroyed civilization. It's not clear to me whether AI is just the final visible piece of our self-destruction, or if it's a brand new thing, hitting us after we were so weakened by the massive changes previously experienced.

I guess I could assign the breaking point still earlier - mass communication and easy air travel are what really broke through the isolated and understandable lives of most people.

I'm pretty sure https://en.wikipedia.org/wiki/Seeing_Like_a_State has bigger lessons than just government or large corporations.

A general problem with optimization and control (in AI and commerce and everything else) is that it limits breadth of activity in (at least) two ways. The obvious way is that it avoids things which oppose the stated goals. That's arguably a good and intentional limit.

But it ALSO avoids things that are illegible or unexpected and hard to quantify. I suspect that's a large majority of experiential value for a large majority of humans.

I don't know if this can be made rigorous enough to generate a 'Friendliness Impossibility Theorem', but it's probably the thing that keeps me up... (read more)

What causes a decision theory to be used?

Dagon

Most (all?) of the decision theory discussion I've seen has been about prediction or contradiction of choice, mostly about how to model the fact that decisions have causes, and those causes could be legible to other agents, or correlated with future experiences in ways that affect the choice under question.

Don't all proposed decision theories just move this down one level, without resolving the underlying contradiction?

It's worth remembering, in discussions about decision theory, deontology vs consequentialism, and related topics, that ALL of our actual examples of agents are MISTAKEN or DECEPTIVE about the causes of their actions.

There are no consistent VNM-rational people.

I wish I could agree-and-downvote (or agree-and-not-vote) on posts. There are a fair few posts that are well-reasoned and probably correct, but just not important or interesting.

I give some probability space to being a Boltzmann-like simulation. It's possible that I exist only for an instant, experience one quantum of input/output, and then am destroyed (presumably after the extra-universal simulators have measured something about the simulation).

This is the most minimal form of Solipsism that I have been configured to conceive. It's also a fun variation of MWI (though not actually connected logically) if it's the case that the simulators are running multiple parallel copies of any given instant, with slightly different configurations and inputs.

Adversarial (SEO) GPT training data?

Dagon

I guarantee someone's thinking about this, but I haven't seen the scammers selling it yet, so I don't know how transparent or discoverable the scraping/input methods are for LLM source data.

Is there any indication that websites or publishers are modifying their pages/data in ways that give themselves more weight in future GPT accessibility/prediction for related prompts?

{M|Im|Am}oral Mazes - any large-scale counterexamples?

Dagon

Dagon, Steven Byrnes

Raemon's recent post reminded me that I'm unsure how best to think about bureaucracies and apparent organizational friction, and that I keep meaning to explore why something about my intuition rebels against the "Maze" framing. I've read a bit of economic and public choice theory, and worked in for-profit organizations very large, large and small, and been exposed to (via friends' work) nonprofits medium and small. I kind of think it's a working equilibrium, that's likely to reoccur or become worse if we disrupt it.

Most people are ... not particularly great at coordination or even surface-level alignment. And not all that motivated to spend more energy than needed on abstract concepts... (read more)

Does a LLM have a utility function?

Dagon

Dagon, cfoster0

There's a lot of discussion and research into AI alignment, almost always about variants of how to define/create a utility function (or meta-function, if it changes over time) that is actually aligned with ... something. That something is at least humanity's survival, but often something like flourishing or other semi-abstract goal. Oops, that's not my question for today.

My question for today is whether utility functions are actually part of the solution at all. Humans don't have them, the most interesting spurs toward AI don't have them. Maybe anything complicated enough to be called AGI doesn't have one (or at least doesn't have a simple, concrete, consistent one).

Is there a worked example of Georgian taxes?

Dagon

I'm trying to think through the effects of a Georgian tax system, where the levy is based on "the value of the land". But in my conception, that's either very small, or it actually includes improvement value (and improvements on neighboring land). The entire value of land is how it's used, I think.

Relatedly, land (and capital generally) is a stock, and taxes are generally a flow. The proposal seems not to be a one-time tax to just take the land value from current owners and then let future owners enjoy the (unfair) gains, so it's something like "imputed rent" or "this year's expected proceeds from using just the land with no labor... (read more)

Believable near-term AI disaster

Dagon

johnswentworth's post about AGI vs Humanity (https://www.lesswrong.com/posts/KTbGuLTnycA6wKBza/) caught my attention in ways that most discussion of AI takeover and the prevention of such does not. Most of the discussion is very abstract, and leaves unstated some pretty deep assumptions about the path and results of smarter-than-human imperfectly-aligned AI. This preserves generality, but interferes with engagement (for me, at least). I think the fundamental truth remains that we can't actually predict the path to unaligned AI, nor the utility function(s) of the AI(s) who take over, so please treat this as "one possible story", not the totality of bad things we should worry about.

40-odd years ago, Douglas Adams wrote:

There is a theory which

... (read 531 more words →)

Laurie Anderson talks

Dagon

Not rationalist per-se, and much more abstract than legible (it is part of a Poetry lecture series), but Laurie Anderson has long been one of my favorite artists, and this is likely of interest (or of enjoyment or some positive experience) to many people here. 10 minutes into the first lecture, she states "We'll start out with two questions. The first question is 'Why do anything?' and the second question is 'How do you know what's good?'"

LESSWRONG
LW

LESSWRONG
LW

For moderately well-resourced people in the US, how quickly will things go from "uncomfortable" to "too late to exit"?

Did the recent blackmail discussion change your beliefs?

{M|Im|Am}oral Mazes - any large-scale counterexamples?

Does a LLM have a utility function?

Dagon

Moral realism - basic Q

What epsilon do you subtract from "certainty" in your own probability estimates?

Should LW suggest standard metaprompts?

What causes a decision theory to be used?

Adversarial (SEO) GPT training data?

{M|Im|Am}oral Mazes - any large-scale counterexamples?

Does a LLM have a utility function?

Dagon

For moderately well-resourced people in the US, how quickly will things go from "uncomfortable" to "too late to exit"?

Did the recent blackmail discussion change your beliefs?

{M|Im|Am}oral Mazes - any large-scale counterexamples?

Does a LLM have a utility function?

Dagon

Moral realism - basic Q

What epsilon do you subtract from "certainty" in your own probability estimates?

Should LW suggest standard metaprompts?

What causes a decision theory to be used?

Adversarial (SEO) GPT training data?

{M|Im|Am}oral Mazes - any large-scale counterexamples?

Does a LLM have a utility function?