mkatavic's Shortform

Marko Katavic

This is a special post for quick takes by Marko Katavic. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Very rough - I am interested in the meta-debate around the ai-debate. I find this important to understand how to cut through with communication and get the public onside.

There's a shocking lack of epistemic humility on all sides of the AI debate, and at all levels of AI expertise. Collapse into binary certainty happens often and without clear labelling of what is informing the collapse. I think the rationalist community (which I a priori class as likely the most epistemically humble, but that's a prior that can get challenged) gets intuitively put in the same hopper by the public - because we can't know the exact probability of x-risk (and against which assumptions?), but we collapse into "we should stop" due to the x-risk being severely asymmetric. To someone who's not heavily epistemically trained, this will feel like an equivalent class of uncertainty collapse.

All sides of the debate are using extremely confident language, and I suspect large parts of each group on each side of the debate on each rung of the proficiency ladder use AI to enhance their thinking, which is a certainty amplifying machine due to its own lack of calibration + sycophancy.

This makes it an impossible debate to resolve - as each side can, at arbitrary times, take out the "hand-waving" accusation - and it'd be true - and therefore the "public opinion heuristic" just depends on what you're reading (or listening) and who reached for it first.

It's like a massively amplifying Donning Kruger across the field getting people into basically theological positions. Meanwhile here we are trying to update people who are entrenching further into a chosen dogma, which is futile.

Am I just discovering water is wet?

1Marko Katavic2mo

Hard to think of examples of the top of my head, might revisit when I encounter in the wild. But actually now that I was trying to think of examples, I can see it as a mixed bag of straw men and conditioning on an uncertain prior without labelling the uncertainty. Maybe the latter is normal in passing language where people assume shared context - and I am just miffed by it because it gets reused when the meme transfers to people who don't have sufficient context to interpret the uncertainty. Example for the latter being when CoT is being talked about in papers as if it holds ground truth on what the model was thinking - that just strikes me as so bizarre to hold as universally true, rather than probably true.

2williawa2mo

Could give an example on lesswrong or a paper where people say that about CoT? The degree to which the CoT is "faithful" is a complicated area people have put a lot of work into studying. I think people on lesswrong are generally careful about which inferences they draw from CoT.

I was primarily talking about the debate outside of LW, where the LW position also gets conflated (as a secondary matter) - I was not trying to make a claim that LW claims suffers from collapsing the uncertainties - apologies if that's how it sounded. I am trying to divide the sphere into:

Rationalist discourse ---Feeds ideas--> AI Safety Mainstream discourse
Vs.
Accelerationist discourse
Vs.
Sceptics discourse

The only thing that refers to LW is that LW also binarises into "stopping AI development is clearly preferable to not stopping AI development" - wh... (read more)