To me this reads as a definition of the term "free will" and then arguing against its existence.

If something can predict your action with better than random accuracy no matter how hard you try to prevent them, you don’t have free will over that action

The word "try" here hides lots of complexity and does some heavy lifting. Can you taboo it and rephrase that? Also, are you rejecting the computational theory of mind?

In the case of the brain-scanner, I’d say “the second before” just reflects the fact that taking an action in the physical world takes >0 time, and once you’ve made the decision to start the action you are no longer free to reverse it

If you take an agent embedded in the environment, I don't think there is a clear-cut boundary between "not yet made a decision" and "have made the decision". This abstraction just breaks down.

My fear with the teleporter has always been the engineering details - can it get a consistent snapshot of me? what about the last moments after the snapshot and before the old copy is destroyed? can it reliably reconstruct me? what happens in case of a failure?

Assuming many worlds quantum mechanics, we should have similar anticipations for forking into two and for tossing a quantum coin.

There is no rollback when open-weight models are almost SOTA. Could we convince people like Zuck that open weights are too risky? I seriously doubt it.

There are some differences between what women consider attractive female traits and what men actually find attractive. The example I like to give is how some women enlarge their lips even though men (usually) don't find that attractive.

I think these are the relevant points:

Maybe women's sense of each other's beauty is more discriminating than men's.

Some women probably don't distinguish a sex symbol's actual physical attractiveness from the other characteristics that would make her rarely and especially appealing to women (like fame, wealth, etc.), and they're neglecting to account for these factors being less salient to men.

This:

Might agent $b$ rewrite agent $a$'s brain to make agent $a$ better satisfy agent $b$'s utility function?  Most forms of wire-heading inherently limit the ability of agents to affect the future

and this

We have not proved that agent $b$ does not try to affect agent $a$'s utility function (in fact, I expect in many cases agent $b$ does try to influence agent $a$'s utility function).

appear to be in conflict. Are you trying to say that depending on the circumstances b may try to influence a's utility function or avoid doing so?

How important is it to keep the system prompt short? I guess this would depend on the model, but does anybody have useful tips on that?

well-established

The usage I'm objecting to started, as far as I can tell, about 2 years ago with Llama 2. The term "open weights", which is often used interchangably, is a much better fit.

At some point the open/closed distinction becomes insufficient as a description. You could very well have an open-source wrapper (or fine-tuning) of something which is closed-source. Just try to not mislead people about what you're offering.

If I vibe-coded an app prompting, say, Claude, and released it along with the generated code, would you have the same objections to me calling it "open source,"

No, because I don't think this misleads people. Granted, the term "open source" is fuzzy at the boundaries. Should we use the term? I don't know, but if we do, it only makes sense if it means something different from "closed source".

wrong in suggesting they prefer to work with the model by editing the training data and "recompiling" instead of starting with the weights

One doesn't exclude the other. If you're creating v2 of your model, you'd likely: take the training code and data for v1; make some changes / add new things; run the new training code on the new data. For minor changes you may prefer to do fine-tuning on the weights.

wildly more expensive 

Suppose I write a program and let people download the binary. Can I say "I spent 100k on AWS to compile it, therefore the binary is open source"?

not even modification

Would you say compiling source code from scratch (e.g. for a different platform) is not a modification?

Even if you're not intending to retrain the model from scratch, simply knowing what the training data is is valuable. Maybe you don't care about the training data, but somebody else does. I don't think "I could never possibly make use of the source code / training data" is an argument that a binary / weights is actually open source.

How does open source differ from closed source for you in the case of generative models? If they are the same, why use the term at all?