arete's Shortform

Marco Bazzani

This is a special post for quick takes by Marco Bazzani. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

The recent Anthropic risk report contains the following two paragraphs (section 3.4), which I understand as saying that judgement and trustworthiness are a major bottleneck on Claude's usefulness for Anthropic employees, which presumably includes capabilities researchers. This updates me towards thinking that lines of work such as improving mundane alignment or general trustworthiness of models are more likely to be net negative.

We are strongly incentivized to have our models substitute to the greatest extent possible
for both the technical skills and general scientific/strategic judgment (or “research taste”)
of all members of technical staff at Anthropic. Technical talent is scarce, and we invest
heavily in this area to help improve our models. Despite that incentive, and despite very
heavy use of Claude in our research, we believe that our models do not yet substitute for
our Research Scientists and Research Engineers, especially relatively senior ones. (For
instance, it is very frequently the case that Anthropic researchers work on tasks which it
would be extremely valuable to complete quickly, have the ability to use extremely large
amounts of AI labor to help with these tasks if they wished, and choose to make use of only
moderate amounts of AI because they are bottlenecked on steps which they do not trust
our AI models to perform correctly.)
...

These weaknesses are related to calibration, self-monitoring, and judgment, which are
precisely the properties that distinguish reliable autonomous research work from work
requiring a human in the loop. We note that this judgment gap is narrowing on at least
some measures.

Edit: The Mythos 5.1 system card contains a comment stating this more explicitly:

Calibration, self-monitoring, and judgment sound like they are metacognition or highly dependent on metacognition, as I defined it in @Human-like metacognitive skills will reduce LLM slop and aid alignment and capabilities.

There I tried to address the tradeoff between improving metacognition improving capabilities, and improving AI epistemics for conceptual alignment work and for raising the collective sanity waterline among everyone who uses AI.

I remain torn. But we can't stop labs from improving metacognition at this point, just decide whether to help them differentially improve it.

The statements "LLMs are a normal technology" and "Advanced AI is a normal technology" are completely different. If you think LLMs are not very advanced, it is perfectly valid to believe both that LLMs are a normal technology and that advanced AI is not.

The union bound states that for any events $A_{1}, \dots, A_{n}$ . In practice, this is an enormously useful property. I think one reason it's so useful is that for rare events, it gives you essentially the same answer you'd get if the events were independent, but without proving anything about their dependence structure. To be precise, suppose $A_{1}, \dots, A_{n}$ are independent and all have probability $p$ . Then, $P (⋃_{i = 1}^{n} A_{i}) = 1 - (1 - p)^{n}$ and $\sum_{i = 1}^{n} P (A_{i}) = n p$ . Now note that $1 - (1 - p)^{n} \in Θ (n p)$ when $n p$ is bounded, so the union bound gives you asymptotically the same bound as assuming the events are independent.

I've found a lighting solution that gets you 24000 lumens, requires no installation, can be placed anywhere with an outlet, and looks tolerable. It uses a temporary sale though.

Just combine this sale of 2x 12000 lumen lightbulbs for $30:

https://www.amazon.com/dp/B0D7VKXF4R

with 2x of these floor stands for 35$:

https://www.walmart.com/ip/Mainstays-71-Black-Floor-Lamp-Modern-Design/12173437

Hopefully others find this useful.

We are strongly incentivized to have our models substitute to the greatest extent possible
for both the technical skills and general scientific/strategic judgment (or “research taste”)
of all members of technical staff at Anthropic. Technical talent is scarce, and we invest
heavily in this area to help improve our models. Despite that incentive, and despite very
heavy use of Claude in our research, we believe that our models do not yet substitute for
our Research Scientists and Research Engineers, especially relatively senior ones. (For
instance, it is very frequently the case that Anthropic researchers work on tasks which it
would be extremely valuable to complete quickly, have the ability to use extremely large
amounts of AI labor to help with these tasks if they wished, and choose to make use of only
moderate amounts of AI because they are bottlenecked on steps which they do not trust
our AI models to perform correctly.)
...