oumuamua

I would assume this is because wasting time (which is to the detriment of your opponent, and which he cannot control) in the first example is a not instrumental to achieving your goal. It is merely a side-effect. "Thou shall not profit from wasting time".

If playing optimally involves making decisions that make the game go longer (such as waiting to draw additional countermagic or whatever), so be it.

That said, I'm surprised Zvi said "match wp" here - I assume this is an oversight on his part. He should just have written "game wp".

I just tried multiplying 13-digit numbers with o3-mini (high). My approach was to ask it to explain a basic multiplication algorithm to me, and then carry it out. On the first try it was lazy and didn't actually follow the algorithm (it just told me "it would take a long time to actually carry out all the shifts and multiplications...", and it got the result wrong.

Then I told it to follow the algorithm, even if it is time consuming, and it did, and the result was correct.

So I'm not sure about the take that

The fact that something that has ingested the ent

If future more capable models are indeed actively resisting their alignment training, and this is happening consistently, that seems like an important update to be making?

Could someone explain to me what this resisting behavior during alignment training looked like in practice?

Did the model outright say "I don't want to do this?", did it produce nonsensical results, did it become deceptive, did it just ... not work?

This claim seems very interesting if true, is there any further information on this?

glamorize

glomarize is the word I believe you want to use.

As a native German speaker I believe I can expand upon, and slightly disagree with, your definition.

I suspect that a significant portion of the misunderstanding about slave morality comes from the fact that the german word "Moral" (which is part of the Netzschean-term "Sklavenmoral") has two possible meanings, depending on context: Morality and morale, and it is the latter which I consider to be the more apt translation in this case.

Nietzsche was really speaking about slave morale. It is important to understand that slave morality is not an ethical system ... (read more)

I don't think the primary decision makers at Nvidia do believe AGI is likely to be developed soon. I think they are hyping AI because it makes them money, but not really believing that progress will continue all the way to AGI in the near future.

I agree - and if they are at all rational they have expended significant resources to find out whether this belief is justified or not, and I'd take that seriously. If Nvidia do not believe that AGI is likely to be developed soon, I think they are probably right - and this makes more sense if there in fact aren'... (read more)

But how would this make sense from a financing perspective? If the company reveals that they are in posession of a 5-level model they'd be able to raise money at a much higher valuation. Just imagine what would happen to Alphabet stock if they proved posession of something significantly smarter than GPT4.

Also, the fact that Nvidia is selling its GPUs rather than keeping them all for itself does seem like some kind of evidence against this. If it were really all just a matter of scaling, why not cut everyone off and rush forward? They have more than enough ... (read more)

Similarly, he claims that the bill does not acknowledge trade-offs, but the reasonable care standard is absolutely centered around trade-offs of costs against benefits.

Could somebody elaborate on this?

My understanding is that if a company releases an AI model knowing it can be easily exploited ('jailbroken'), they could be held legally responsible - even if the model's potential economic benefits far outweigh its risks.

For example, if a model could generate trillions in economic value but also enable billions in damages through cyberattacks, would re... (read more)

1, Yes, but they also require far more money to do all the good stuff as well! I’m not saying there isn’t a tradeoff involved here.

2, Yes, I’ve read that. I was saying that this is a pretty low bar, since an ordinary person isn’t good at writing viruses. I’m afraid that the bill might have the effect of making competent jailbreakable models essentially illegal, even if they don’t pose an existential risk (in which case that would be necessary ofc.), and even if their net value for society is positive, because there is a lot of software out there that‘s ins... (read more)

Correct me if I'm wrong, but it seems to me that something this law implies is that it's only legal to release jailbreakable models if they (more or less) suck.

Got something that can write a pretty good computer virus or materially enable somebody to do it? Illegal under SB1047, and I think the costs might outweigh the benefits here. If your software is so vulnerable that an LLM can hack it, that should be a you problem. Maybe use an LLM to fix it, I don't know. The benefit of AI systems intelligent enough to do that (but too stupid to pose actual existent... (read more)

How is this not basically the widespread idea of recursive self improvement? This idea is simple enough that it has occurred even to me, and there is no way that, e.g. Ilya Sutskever hasn't thought about that.

Don't do this, please. Just wait and see. This community is forgiving about changing ones mind.

Some hopefully constructive criticism:

• I believe it's "agentic", not "agentive".
• "Save scumming" isn't a widely known term. If I hadn't known exactly where this was going, it might have confused me. Consider replacing it with something like "trial and error".
• I would rework the part where the blob bites the finger off, it causes people to ask stuff like "but how should a piece of Software bite my finger?", this derails the conversation. Don't specify exactly how it's going to try to prevent the pushing of the button, explain that it has a strong inventive

While I am not a lawyer, it appears that this concept might indeed hold some merit. A similar strategy is used by organizations focused on civil rights, known as a “warrant canary”. Essentially, it’s a method by which a communications service provider aims to implicitly inform its users that the provider has been served with a government subpoena, despite legal prohibitions on revealing the existence of the subpoena. The idea behind it is that it there are very strong protections against compelled speech, especially against compelled untrue speech (e.g. up... (read more)

I believe Zvi was referring to FAAMG vs startups.

I read A Fire Upon the Deep a few years ago, and even back then I found it highly prescient. Now I'll take this sad event as an opportunity to read his highly acclaimed prequel A Deepness in the Sky. RIP.

Murder is just a word. ... SBF bites all the bullets, all the time, as we see throughout. Murder is bad because look at all the investments and productivity that would be lost, and the distress particular people might feel

You are saying this as if you disagreed with it. In this case, I'd like to vehemently disagree with your disagreeing with Sam.

Murder really is bad because of all the bad things that follow from it, not because there is some moral category of "murder", which is always bad. This isn't just "Sam biting all the bullets", this is basic util... (read more)

To me it feels exactly like the kind of habit we should get into.

Imagine an advanced (possibly alien) civilization, with technology far beyond ours. Do you imagine its members being pestered by bloodsucking parasites? Me neither.

The existence of mosquitoes is an indictment of humanity, as far as I'm concerned.

Is there an actually good argument for why eliminating only disease carrying mosquitoes is acceptable, rather than just wiping them all out? There is no question that even without the threat of malaria, creatures like mosquitoes, bed-bugs and horse-flies decrease the quality of life of humans and animals. Would the effects on ecosystems really be so grave that they might plausibly outweigh the enormous benefits of their extinction?

You know the way lots of people get obsessed with Nietzsche for a while? They start wearing black, becoming goth, smoking marijuana, and talking about how like “god is dead, nothing matters, man.” This never happened to me, in part because Nietzsche doesn’t really make arguments, just self-indulgent rambles.

This is objectionable is many ways. To say that one of the most influential German philosophers produced only self-indulgent rambles is a sufficiently outrageous claim that you should be required to provide an argument in its favor.

I don't even disag... (read more)