This:

Might agent $b$ rewrite agent $a$'s brain to make agent $a$ better satisfy agent $b$'s utility function?  Most forms of wire-heading inherently limit the ability of agents to affect the future

and this

We have not proved that agent $b$ does not try to affect agent $a$'s utility function (in fact, I expect in many cases agent $b$ does try to influence agent $a$'s utility function).

appear to be in conflict. Are you trying to say that depending on the circumstances b may try to influence a's utility function or avoid doing so?

How important is it to keep the system prompt short? I guess this would depend on the model, but does anybody have useful tips on that?

well-established

The usage I'm objecting to started, as far as I can tell, about 2 years ago with Llama 2. The term "open weights", which is often used interchangably, is a much better fit.

At some point the open/closed distinction becomes insufficient as a description. You could very well have an open-source wrapper (or fine-tuning) of something which is closed-source. Just try to not mislead people about what you're offering.

If I vibe-coded an app prompting, say, Claude, and released it along with the generated code, would you have the same objections to me calling it "open source,"

No, because I don't think this misleads people. Granted, the term "open source" is fuzzy at the boundaries. Should we use the term? I don't know, but if we do, it only makes sense if it means something different from "closed source".

wrong in suggesting they prefer to work with the model by editing the training data and "recompiling" instead of starting with the weights

One doesn't exclude the other. If you're creating v2 of your model, you'd likely: take the training code and data for v1; make some changes / add new things; run the new training code on the new data. For minor changes you may prefer to do fine-tuning on the weights.

wildly more expensive 

Suppose I write a program and let people download the binary. Can I say "I spent 100k on AWS to compile it, therefore the binary is open source"?

not even modification

Would you say compiling source code from scratch (e.g. for a different platform) is not a modification?

Even if you're not intending to retrain the model from scratch, simply knowing what the training data is is valuable. Maybe you don't care about the training data, but somebody else does. I don't think "I could never possibly make use of the source code / training data" is an argument that a binary / weights is actually open source.

How does open source differ from closed source for you in the case of generative models? If they are the same, why use the term at all?

There is the possibility of misgendering somebody and them taking it seriously. Sometimes it feels like you're walking in a minefield. It's not conducive to a good social interaction.

too few pronouns, and communication becomes vague and cumbersome

I'm wondering why languages like finnish can do just fine with "hän" while english needs he/she.

French to English you always translate as "you". You probably mean translating from English to French where you need to make a judgement whether to use "vous" or "tu".

I think of preferences as a description of agent behavior, which means the preferences changed.

When you say "got better at achieving it's preference" I suppose you're thinking of preference as some goal the agent is pursuing. I find this view (assuming goal directedness) less general in its ability to describe agent behavior. It may be more useful, but if so I think we need to justify it better. I don't exclude the possibility that there is a piece of information I don't know about.

Goal-directedness leads toward instrumental convergence and away from corrigibility. If we are looking to solve corrigibility, I think it's worth it to question goal-directedness.

• vim
• closing stuff - window/application list menu which opens with alt+tab / command+tab; dropdown menus; popup messages in sites