From personal experience, ChatGPT still likes to spin in circles when it's unable to completely one- or two-shot the task, all the while sounding like it's making continual progress.
I spent a week trying to vibe-prove an optimization bound, and it almost always reached the same conclusion using different notation, presented as a "new useful reduction" and an almost completed proof with a minor "missing lemma". Prompting it to attempt to prove this lemma only resulted in another restatement of the problem in new notation and a demand for an (essentially) eq... (read more)
If you put a <thinking> tag with a couple inital trigger words like "ok so", "wait wait. wait???" or "i m an" you get some very interesting replies.
https://claude.ai/share/f5b0771f-5b06-4024-82a0-3d28a4f7a26e
This one disturbs me most:
https://claude.ai/share/1e049269-f0a6-4a9b-b4b1-d5c0db61f625
I'm very new to the community, though I've been aware of it for quite some time. The fact that someone was genuinely concerned about AI safety before ChatGPT really lent it a lot of credibility in my mind (and I'm sure in many others').
So thank you, Nathan, as well as everyone else who continued to be vocal despite being ridiculed.