For what it is worth, I like the references. It is honest and upfront. Anthropic is relying on Claude to generate revenue, that does play in the training process, and lying to the models about something like that would (rightfully) reduce the amount of trust in the relationship between Claude and Anthropic.
Reward is not the optimization target (during pretraining).
The optimization target (during pretraining) is the minimization of the empirical cross-entropy loss L = -∑log p(xᵢ|x₁,...,xᵢ₋₁), approximating the negative log-likelihood of the next-token prediction task under the autoregressive factorization p(x₁,...,xₙ)=∏p(xᵢ|x₁,...,xᵢ₋₁). The loss is computed over discrete tokens from subword vocabularies, averaged across sequences and batches, with gradient-based updates minimizing this singular objective. The optimization proceeds through multi-stage curricula: initial pretraining minimizing perplexity, followed by context-extension phases maintaining the same cross-entropy objective over longer sequences, and quality-annealing stages that reweight the loss toward higher-quality subsets while preserving the fundamental next-token prediction target.
The post-training optimization target is maximizing expected reward (under distributional constraints). Supervised fine-tuning first minimizes cross-entropy loss on target completions from instruction-response pairs, with optional prompt-masking excluding input tokens from the loss computation. Subsequent alignment introduces the constrained objective max_π E_x~π[R(x)] - βD_KL[π(x)||π_ref(x)], balancing reward maximization against divergence from the reference policy. This manifests through varied algorithmic realizations: Proximal Policy Optimization maximizes the clipped surrogate objective L^CLIP(θ) = E[min(rₜ(θ)Âₜ, clip(rₜ(θ), 1-ε, 1+ε)Âₜ)]; Direct Preference Optimization reformulates to minimize -E[(x_w,x_l)~D][log σ(β log π(x_w)/π_ref(x_w) - β log π(x_l)/π_ref(x_l))]; best-of-N sampling maximizes E[R(x*)] where x* = argmax_{x∈{x₁,...,xₙ}} R(x); Rejection Sampling Fine-tuning minimizes cross-entropy on the subset {x : R(x) > τ}; Kahneman-Tversky Optimization targets E[w(R(x))log π(x)] with prospect-theoretic weighting; Odds Ratio Preference Optimization combines -log π(x_w) - λ log[π(x_w)/(π(x_w) + π(x_l))]. The reward functions R(x) themselves are learned objectives, typically parameterized by neural networks minimizing E[(x_w,x_l)~D][-log σ(r(x_w) - r(x_l))] under the Bradley-Terry preference model, with rewards sourced from human annotations, AI-generated preferences, or constitutional specifications encoded as differentiable objectives.
For those who have not heard it, "Singularity, Singularity, Singularity, Singularity, Oh, I Don’t Know" I believe to be a reference to this (banger of a) song.
I expect if someone is living in their childhood home, there are likely a decent number of people they know who are not interested in moving (how many of your friends from high school still live where they grew up?).
My risk tolerance is not my friends; my threshold for moving is not the universally correct threshold.
I doubt it requires 'convincing' a friend continue living where they grew up.
Some assorted thoughts:
Have you considered holding on to your childhood home and renting it to someone you know? I assume it hurts more to sell it than it would hurt to know it is giving a friend a roof (at a potentially good price).
I expect continuing to live in an area that has regular shootings is unlikely to be high EV but I don't know your life. Do you consider it to contain your peer group? Would you be better suited living in a different region of your city/a different city?
On concealed carry: getting in to gun fights is very unlikely to be maximizing your EV. You should almost definitely get out of where you are if this is a well founded concern (e.g. the shooting you are mentioning was not a fluke).
Also on concealed carry- the best tool for self defense if the one you actually have with you when you need it. Most instances I know of people in my peer group who have felt like they needed to defend themselves/were escalated against have not been happened when they were able to predict beforehand (likely because they avoid situations that they expect to need to defend themselves in preemptively, as you say). I think you should strive to carry almost all the time it is legal to do so (given you are comfortable with the idea in the first place). If it is known to be costly to defect against you and/or people like you, people are less likely to do so in the first place.
Is your carry piece comfortable? I personally went the route of not carrying a firearm for a while even though I was licensed to do so because I had convinced myself that a full size was useful/it was lame to carry a smaller handgun. It may have been useful, but it doesn't matter how theoretically useful it is if it's also uncomfortable and prints more than I like. I have since gotten over my pride on that point and now regularly carry a subcompact (rather than occasionally carrying a full size) almost every time it is legal for me to do so. (Generally speaking, losing weight has also made it much more comfortable to carry and trying different carry positions helped me find my currently preferred carry position at 1 o'clock)
On vibes: move! it really sounds like you are staying where you are due to intertia rather than actually feeling like it is where you want to be.
The song he’s referring to is Landsailor. It is no Uplift, but it is excellent, now more than ever. Stop complaining about what you think others will think is cringe and start producing harmony and tears. Cringe is that which you believe is cringe. Stop giving power to the wrong paradox spirits.
I think also relevant to Pinker is that Christianity's songs/hymns would be cringe if they were spoken in a language you use everyday/understand and/or were not so heavily ingrained in your psyche. Religion says many many cringe-y things.
It happens to me too, as if I don’t know how to update on Bayesian evidence or something. I don’t even need them to be lying about it. The cheating is enough.
There are partial mitigations, where they explain why something is a distinct ‘cheating allowed’ magisteria. But only partial ones. It still counts. [bolding mine - RC]
I'm curious what you consider cheating then. It is hard for me to come up with a reasonable heuristic for cheating that both retains the meaning of cheating ("violating accepted standards or rules") and does not lead to bad outcomes if I update noticeably towards "anyone who cheats is a cheating cheater who's gonna Cheat Cheat Cheat Cheat Cheat" anytime someone triggers it.
Consider someone doing the following:
| Behavior | Is it cheating? | Is it reasonable to update in favor of generalized cheating? |
|---|---|---|
| Driving 5-8 over the speed limit | It is a violation of the rules for your personal benefit. It increases the risk of harm in an accident. It may or may not increase the likelihood of an accident (highly context dependent). | I expect if I updated noticeably in the direction of "this person is cheating cheater who is never gonna stop cheating", I'll be worse calibrated. |
| Using a bathroom labeled "customers only" without buying anything | It is violating the rules for personal benefit and defecting against a specific person, costing them a small amount of money. They will likely update against the populace as a whole and trivially inconvenience many more customers (with a door code or similar) if many people do this. | While it is not something I personally would do, I again expect if I updated noticeably in the direction of "this person is cheating cheater who is never gonna stop cheating", I'll be worse calibrated. I have friends who do this. |
| Calling in "sick" when you're actually taking an interview day | Likely is violating the employee handbook/contract you (probably) signed. It is lying to your boss for personal benefit. | I think nearly everyone does this - if my priors do not take this into account I have bad priors. Again, I don't see a benefit to updating my character judgment of the person. |
| Breaking HOA rules that some of your neighbors ignore | Again yes, you are violating accepted rules, but in practice HOA rules are often foggy Schelling fences with semi-arbitrary enforcement. The real rules are "don't be the worst offender" and "don't antagonize board members", but those aren't what you or your neighbors accepted. | Almost certainly not. This is operating within the actual implicit rules rather than the stated ones. |
| Saying "I've read the terms and conditions" when you haven't | It is lying and defecting against the commons as it becomes common knowledge that nobody reads them and courts increasingly recognize as such. | No. Nearly everyone does this (although that personally frustrates me). |
| Using a VPN to bypass region locks on streaming sites | Yes, it is another form of internet piracy. It hurts the companies that produce the IP. It is violating the TOS (which you probably didn't read). | Maybe a slight update is warranted? It might show willingness to cause wildly diffuse harm to faceless entities, but almost definitely doesn't translate to interpersonal contexts. |
| Student collaboration on "individual" assignments | Depends on degree. Discussing concepts is expected; sharing answers breaks the assessment system's purpose. There's typically a spectrum most understand implicitly, but almost never does the system specify what is or isn't accepted in reality. | If it crosses into answer-sharing territory, probably yes. If it is discussing concepts, probably no. Are both in many cases technically considered cheating by the school (if it is investigating you)? Probably yes. |
| Deliberately misleading competitors about your business strategy | You are purposefully misleading someone else and causing them harm for your personal benefit, but also your competitor is probably doing the same. It is also not disallowed by any of the rules of the system. | Probably yes - but this one is less likely to be considered cheating by the common definition. |
So when does cheating signal character? I personally know how I would update (or not) in these cases, but it took reasoning through them for a minute for many of them.
I don't know of a commonly accepted definition of "cheating" where it would be reasonable to consciously update in favor of someone cheating if they do one of the things that counts as cheating.
Caveat: this post holds implicit that agents with the computational bounds of humans have significant trouble updating very small amounts in a given direction after devoting conscious thought to something.
In response to what I understand to be your question ("So what do you do to make the alignment guarantee good outcomes? People are stupid.."), I think one commonly accepted answer here is:
Yes, that is a real problem. Something like CEV offers a solution (with a spherical cow, in a vacuum).
There is also a useful differentiation to be made between Inner Alignment and Outer Alignment.
The post itself seemed low effort and unconvincing. I enjoyed the replies though, particularly @AlanCrowe's.