Wouldn't the equilibrium here trend towards a bunch of wasted labor where I deliberately lowball the value of the land, and then if someone offers a larger amount, I just say no and then start paying the larger amount, thus having a potential to pay less tax but losing nothing if I'm called out for it? No downside to me personally, and if this became common, it'd be harder to legitimately buy stuff. Seems like you'd need to pay some sort of fee to the entity credibly offering this larger amount to make it worth it.

This is the kind of content I keep coming back to this site for.

• Obviously correct
• Immediately useful as a day-to-day habit of thought
• "Why didn't I think of that!?"

I also like that it's practical and practicable in day to day life while also being important for bigger, important questions.

Another example to add to the post. A few years ago I was learning to play Go, and thus playing against superior players. In Go, typically what happens is one player will make a move that signals something, like "I can take this territory" or "I can attack this group of pieces successfully" and the other player needs to decide whether to directly oppose this move, or to accept it and play elsewhere instead.

So I asked myself - if I disagree with a superior player's assessment of the situation (i.e. he makes a move I think I can punish), what should I do? Statistically I'm probably wrong about my assessment. But then I realized that if I accepted their move, I would never find out why they were right. If I opposed the move, and was quickly punished (my group was destroyed, or the player successfully defended their "too aggressive" move) I would get to find out why I was wrong immediately. So I began playing according to my instincts even though they would lose me more exchanges in the short term, since it was the direction of maximum learning.

Outside of the office, I generally find it difficult to get appreciable amounts of work done. It feels like it takes a significant exertion of willpower to go from not doing a work-related task to doing one, at which point it generally becomes easier to continue with that task for a while. Performing this mental motion a few times per workday is enough for me to get close to full time hours in, but doesn't feel enough for sixty hours. If I don't perform this mental motion successfully, I wind up in a state of internal tension where I'm not actually putting in consistent mental effort towards solving the next problem in front of me. I do have a fix for this already - I work better in an office, where everyone around is also working.

So, the natural solution here would be "Find yourself an office type environment doing valuable work where it is entirely normal and expected for people to work these kind of hours." This runs into the second issue, which is that on the rare occasions I have worked 10+ hours in a day, or generally pushed myself harder to try and stretch the edge of my conscientiousness, I tend to get headaches. Which are both unpleasant and also reduce my productivity, which ruins the entire point of the exercise. (The headaches are a known problem - I'm on a headache preventer that minimizes them. I could try upping the dosage, but I've already been told by one doctor that I should probably try not to be on this medication indefinitely.)

This also means that it wouldn't be a good idea to go to an environment where everyone's expected to work 60 hours, if I then don't end up being able to do that even with the social and logistical environment set up in my favor. So it'd have to be an environment where it was both normal and expected to work 60 hours AND to work 40, AND the work was object-level valuable in my opinion, to be worth trying this experiment. This could be possible - I haven't actually tried to seek out such an environment. But I do notice that I still do anticipate failing the original 60-hour goal due to the problem in the second paragraph.

On thinking about them:

"Having both the motivation and the mental stamina to work 60-hour weeks reliably." Actually this probably would be "hard" rather than "impossible". There are things I can try here that I haven't tried that might work, so I have not yet tried enough things to declare it impossible. It's more like I anticipate the possibility of this being impossible, as opposed to actually considering it impossible. Not a good example.

"Gaining 15+ points of IQ or the thing that IQ is measuring." - So it seems like there are two ways this problem could go. In World A, there exists some stack of exercises and nootropics that can already let me do this without sacrificing something I'm not willing to sacrifice. In this case the problem is that of finding it - people can bullshit or lie, supplements are not a field where I expect complete honesty, and it wouldn't surprise me if there's a high amount of individualism such that someone could truthfully say they gained 15+ IQ points on X, but I need Y instead to achieve the same outcome. If I had to try each combination myself I'd run out of time very quickly. It's impossible in the sense of winning the lottery is impossible - not something I can reliably make happen, as opposed to literally can't be done. This is close enough that I consider it to be equivalent.

Alternatively, maybe this stack just doesn't exist at all. Nootropics and exercises will not get you there. In which case, solving the problem means making advances in cognitive science that our culture hasn't yet figured out how to do. And it's not clear to me how I would succeed where lots of others have failed, here - leading onto my third point, I don't see how I have a comparative advantage in this area, and if I have to work full-time for years to get to this point it is no longer worth it, when the point is to make me more effective at solving my current chosen problem.

"Taking something that isn't my comparative advantage and making it that way" - I can think of a bunch of actions I can take here that would let me do better than I currently am. Getting tutoring, improving my ability to learn, talking it over with more experienced practitioners, etc. The key impossibility here is that people who are better than me at this can just do this stuff too and probably already are.

Like, imagine I want to be a top mathematician since I'm convinced that's the only skill worth knowing for alignment. I can hire a tutor (and have done so), but better mathematicians can also do this and probably have. How do you get good enough to meaningfully contribute when people exist in this field who did the IMO at 17 and can do any of the same improvements I can come up with? So, I would have to find some method that was A) Incredibly effective, enough to bring me up to par with more talented people and B) Other people are unable or unwilling to do this thing, even really talented people in the field.

I don't know how much context you need for the more personal examples, so I figure I'll give them without context and then if you need more you can ask:

• Having both the motivation and the mental stamina to work 60-hour weeks reliably.

• Gaining 15+ points of IQ or the thing that IQ is measuring.

• Becoming good enough at a field (abstract mathematics, mechanistic interpretability) that I've previously tried and found myself to be not that good at or interested in, such that it would then be worth my pursuing it as part of a research direction. (Another way you could write this is "Taking something that isn't my comparative advantage and making it that way")

• Convincing an arbitrary person my values are good and worth adopting within a sixty-minute conversation. (I don't actually WANT to do this in full generality! But it sure does seem impossible and it sure would be nice to do things that are closer to that area, sometimes)

• Becoming a professional tennis player at the age of 33 with little tennis experience. (Again not something I actually want, but it sure does seem impossible. I figure if this is a bad example, you can just not include it.)

Unrelated to the main point, but - where is this second Merrin story found? I really liked the first one, but didn't know there was a second one, and glowfic authors tend to have a lot of story threads.

I am reminded of the Message for Garcia essay as a pretty striking example of this.

One would hope indeed. But even so, we do now know that this is likely to be the kind of action that could be detected and opposed. And since I didn't predict in advance that this would happen, especially at this capability level, the update for me is that it's going to be significantly harder than I would hope.

I recently read Anthropic's new paper on introspection in LLM's. (https://www.anthropic.com/research/introspection)

In short, they were able to:

• Extract what they called an ALL CAPS vector, the difference between a prompt with ALL CAPS and the same prompt without all caps.
• Inject that vector into the activations of an unrelated prompt.
• Ask the model if it could detect an injected thought. About 20% of the time it said yes, and identified it with something like "loud" or "shouting" which is quite similar to the all caps they were going for.

They did this with a bunch of concepts. And Opus 4 was, of course, better than other models at this.

To give some fairly obvious thoughts on what this means:

• If a model can unreliably do this now when directly prompted to do so, I expect the model can reliably do this without being prompted in around 2-3 years from now.

• If you have an alignment plan that involves altering the model's thoughts in some way, consider that the model will probably be aware by default that you did that, and plan accordingly.

Finally a more personal thought:

The idea of "Well, if you try to adjust a superintelligence's thoughts, it will know you did that and try to subvert it / route around the imposing thoughts" would have sounded weird to me, if I'd tried saying it out loud. We're talking about literally altering the model's mind to think what we want it to think, and you still think that's not enough for alignment? I note that I did think that, but I was thinking in terms of "The concepts you are altering are a proxy for the thing you are actually trying to alter, and the important performance-enhancing stuff like deception will migrate to the other parts" and not "The model will detect you altering its mind and likely actively work against you when it builds the situational awareness for that". And yet, here we are, with me insufficiently paranoid. Unfortunately, reality doesn't care about what sounds intuitively plausible.

I wonder if this is an example that would allow people to feel, viscerally, the difficulty of aligning an actually smart entity. Here we have a capability that sounds scary and superhuman and only arose with smart models. The idea of a model being aware of when you're altering its thoughts is terrifying. The lesson we would like to give to decision-makers is something like:

"AI's will soon know when you try to adjust its thoughts. That's how smart these things are. We can employ a primitive type of mind control on them and that still isn't enough. We are really not ready for even smarter systems than this. Any alignment plan that might work has to be stronger than mind control. Anything below the level of literal mind control isn't even the ante to sit down at the table any more."

I can't literally reduce this down to five words, but a succinct message might be something like "Mind control won't work. We need a plan that's better than that."