Dweomite

Even if August is on the level, humans are still losing most of the light-cone. (How bad that will seem depends on what you think the alternative was.)

On my models, the most likely reason for it to hurry would be to reduce the chances that some other AI becomes strong enough to fight it.

Hoping to capture a larger piece of the universe also seems like a plausible motive.

Worry that they might be discovered might be a motive to hurry the deployment of the ETC and other parts of the control system, but once those are in place it would think it's effectively safe from humans even if it waits on the public announcement.

If you want a better understanding of what you mean by novel game concepts and whether they exist, my suggestion is to look for examples of real historical games that would meet your bar.

For example, does Magic: The Gathering meet your bar? I consider it an unusually innovative game for the collectible angle.

I'd also note that whether a concept "looks" novel can change a lot depending on your level of abstraction. In some sense, if you already have a Turing-complete language, then all computable concepts are just combinations of known building blocks. Simi... (read more)

I did not pick up from the OP that there was a plan for costs to be lower to posters who had previously proved themselves, or for there to be any way for posters to make money at all, so there might be some important pieces in your head that didn't make it into the post. (Or I missed them.)

My immediate question is "what are the posters hoping to gain, that they're willing to pay to have their posts published?" Is this for charity? For prestige? For cash prizes? What exactly is the value you're offering to the people you are hoping will pay you? I suspect that how the service would need to be structured, and how likely it is that you could get people to use it, would depend significantly on this answer.

Additionally, I'm skeptical of the idea that moderators will gain reputation organically.

If you reward them for endorsing things that are late... (read more)

Not very long ago (maybe the 1970s), poverty was still quite common. (And somehow, my impression is that kindness/virtue mattered much more back then. My take is that people are relatively similar in terms of economic situation, and there is sense of "we can do this better together".)

I haven't verified this, but I'm reminded of a claim I once read that cultures arising in regions of scarcity (e.g. deserts) tend to develop strong traditions of hospitality and helping people, because you could literally die if you traveled somewhere in a desert and then coul... (read more)

Someone I know (let's call him Alex) once told me that his friends had reported that when Alex is engrossed in something (typically a video game), Alex will respond to questions in a normal-sounding but useless way without breaking his focus. He recounted one incident that went something like:

Robert: Hey Alex, where is the seamstress character in this game?

Alex: I don't know where the seamstress is.

Robert: Really? You seem to know where all the key characters are.

Alex: Well, I don't know where the seamstress is.

Robert: ...wait, your character has the tailo... (read more)

I notice in your negative example that Bob hasn't only failed to give an invariant, but has failed to say anything at all about the nature of the difficulty that he believes is essential.

That broadly seems like a reasonable response, but it seems equally reasonable even if you had not asked them to express their problem as an invariant. Insofar as this works, it seems like it works by asking for rigor, not by asking things to be expressed as an invariant.

I'm not sure I followed that. Are you saying something like: "Even for fallible humans, it seems likely there exists some argument good enough to persuade them of the truth, if you could somehow find that argument"?

If "there is no bell on the cat" count as an "invariant", then I'm confused about what this proposal rules out. For any alleged cat-belling problem P, what stops you from picking an "invariant" that is just a paraphrase of the problem, like "P has not happened"?

If they say that their invariant is "no monkey gives birth to a non-monkey", then how are you any better off than before you invoked this rule?

Seems to me there's some tension between "you can't tell a model to go do some hacking and then get mad about the specific hacking it does" and "a paperclipper would never use you for raw materials because that's obviously not what you meant".

This information presumably wasn't all available at the time of your class, but if I were explaining the Hugging Face incident now, some things I would emphasize (drawing mostly from here):

• About 1200 different AI instances were being trained in isolation from each other
• They broke out of isolation and created a message... (read more)

I don't think this affects your central point, but I'd expect a fairly large class of simulations to have exactly one miracle in them, and I don't think your argument weighs against this.

For example, someone with a question like "what if technology X had been discovered 100 years earlier?" might run a simulation with exactly one miraculous intervention to cause technology X to be discovered at the time of the simulator's choosing. Or "what if WW2 had ended differently?" could lead to a simulation with exactly one miraculous intervention to change the outc... (read more)

When you are counting up the total value being produced in a free labor market, I believe you are implicitly including the part of that value that is captured by the laborers. That is, if I pay you $10 to produce a good worth $20 to me and so two people each end up $10 richer, you are counting that as $20 worth of production. Under this accounting, I agree it would be quite surprising if slavery were efficient.

But surely no slavery advocate was accounting that way? The efficiency seems more doubtful if you only care about a fixed group of people that doesn... (read more)

I think your description of how LLMs receive all their input in a single stream does a good job of explaining the issue but overstates the difference compared to a human.

Your examples of how humans are different focus on our internal thoughts and memories. I agree that these are privileged at a hardware level, but I think they're the exception. Instructions from your boss, your client, your child, and a stranger all arrive via the same channels, and you still need to treat them differently. And there are effective attacks in the wild that rely on confusion... (read more)

even when models notice spoofed prefills, the attack can still participate in computation.

Also true for humans; attacks that the human successfully recognizes as an attack can still affect the human's subsequent reasoning. For example, the anchoring effect works even when the anchor is blatantly wrong; blatant advertising tricks can still work against humans who know how they work; etc.

This isn't especially relevant, but I've started to occasionally notice that some snippet of song lyrics unintentionally could be interpreted as being about rationalsphere ideas if taken completely out of context, and your comment reminded me of one such:

But can you make the difference?
Are endings set in stone?
A band of wayward strangers
can't stand up to gods alone
So what could you become, then—
But not lose who you are?
And will you stop before
you go too far?

Make your move and change it all
Forevermore

(From Make Your Move by Aviators; there's 2 slightly dif... (read more)

As I read the OP, I thought to myself: If I were to steelman the people the post is complaining about, I would guess that they are interpreting the complaint about the problem as an implied proposal for how to address the problem, and they are reacting to the perceived-implied-proposal.

It seems like you're thinking along similar lines, but are about 3 assumptions further down that road:

• Not only are you guessing that they might have thought this, you're pretty sure this is what they were thinking
• Not only are you sure they thought this, you're sure they were... (read more)

Alternate steelman - they're worried that you're intending to misleadingly quote their answer in a different context, and have rigged the question to get the quote you want.