rcc

• For questions requiring small integers (6 domains). One fixed list of 255 consecutive integers covering the full possible answer range.
• For large/ unbounded answers, 255 consecutive integers containing the answer at a random position.

What's the value of assessing Jev on those types of questions? Those are clearly not the use cases it was designed for.

It feels very weird to test the model on capabilities it doesn't claim to posses, then draw a conclusion that it "has a very jagged frontier". If you asked Claude to generate photorealistic images^[1] you could ... (read more)

It's not only about which risks get prioritized. It's also about who is advocating for that. For most leftists, the AI labs are firmly at the top of the existing power structure, same as all large corporations. AI labs are currently advocating to focus on X-risks. When someone I believe to be acting against my best interests is telling me something, it's very natural to reject that view by default.

While I personally agree that the EU doesn't have much leverage in preventing the doomsday scenarios, a "young talented graduate" also doesn't, no matter where they are.

But just preventing the doom is not where the journey ends. Lots of work will be needed for AI to go right across the board. Working at a relevant EU institution doesn't seem like the worst way to contribute.

I think the unintuitive part is that AIs care a lot less about the potential negative consequences than humans. We do have a lot of stories and cultural tropes about people hurting others to get money or power - that is intuitive but it's not really the sort of instrumental convergence that is relevant for most AI discussions today.

We do not have stories and tropes about humans hacking corporations and governments for the sake of doing better on an exam^[1]. For an average person this sort of behavior doesn't make sense, even if the goal is instrumental. Th... (read more)

It's hard to run a business as an amnesiac, even if you keep good notes.

With the physical limitations of a human, yes. It's a different story if you could read and write those notes instantly at any time. I'd expect AI to still benefit from internalizing learning but I doubt that the penalty of not doing so is sufficient for humans to maintain competitive advantage.

Out of curiosity - why such a long delay between collecting the data and publishing the results?

Who, except maybe rationalists, would believe (or give money to) the Trustworthy Institution rather than an Ideological Institution Supporting Their Worldview? I don't think there's a market for it.

Relevant xkcd:

tbh I don't understand your point. I feel like you're trying to gesture at an internal contradiction in my post against things I did not say? Which feels like an odd standard to be held to.

I assume you are talking about this:

You say that "no current publicly available model is known to use neuralese, and the theoretical benefits have not really been demonstrated or realized." At the same time you make a pretty compelling argument that Astra is using neuralese.

I agree that it's technically not a contradiction - your claim is about certainty and your argumen... (read more)

There's a major difference between the two.

AI Safety went mainstream because it reinforced a preexisting bias against AI - people afraid for their jobs, tired of slop and convinced that it's bad for environment. There was already a significant (even if highly fragmented) anti-AI movement before the general population started seriously considering x-risk.

Longevity won't be benefitting from the preexisting bias against death. Culturally there's no such thing. We don't have people protesting against death and writing articles about how bad death is.

What we ha... (read more)

Nvidia is approximately as involved in the overall dynamic.

Is it, actually? GPUs are more or less a commodity these days. One that is used for both good purposes (e.g. current LLMs which contribute to economic output) and bad purposes (i.e. training new AIs that might pose danger).

What would the defectors advocate for?

Even if NVIDIA wanted to stop the flow of new GPUs to the "bad purpose" use cases, they wouldn't be able to do so. Only the US Government could actually enforce such decision.

Stopping the production of GPUs entirely would be both illegal (ac... (read more)

Some level of swarm behavior will always be necessary for some use cases due to pure physics. If your compute is spatially distributed (e.g. placed into physical robots) then the bandwidth available between compute instances will typically be significantly lower and less reliable than the bandwidth available within a compute instance.

For an easy example, just imagine a swarm of military UAVs attacking an enemy compound. You can only pack so much compute into a single UAV and the quality of the data link between any two UAVs will vary dynamically, including... (read more)

As I understand this proposal boils down to creating more auditable artifacts during the operation of an AI system. I'm not sure we actually need more auditable artifacts today. We already have conversation logs, CoT and tool call logs available. All recent incidents could have been detected from those artifacts alone. The gap appears to lie primarily in the willingness of frontier AI labs to spend resources on real-time detection and share the findings, not in the lack of data to analyze.

I fully agree with the claim that moving away from CoT is bad but I think that your claim that it's for "dubious benefits" is quite weak and counterproductive to the overall argument.

You say that "no current publicly available model is known to use neuralese, and the theoretical benefits have not really been demonstrated or realized." At the same time you make a pretty compelling argument that Astra is using neuralese. We also ~know^[1] that Astra is notably better than any previous model. Sure, correlation does not imply causation but in this case it does ... (read more)