Yesterday, I couldn't wrap my head around some programming concepts in Python, so I turned to ChatGPT (gpt-4o) for help. This evolved into a very long conversation (the longest I've ever had with it by far), at the end of which I pasted around 600 lines of code from Github...
This is a blogpost version of my AI Safety Fundamentals project report. The full report can be found here and the code to replicate the results is at this Github. Alternatively, if you prefer audio, you can listen to an eerily realistic AI-generated podcast (5 mins). I will be grateful...
TLDR: In response to Leopold Aschenbrenner’s ‘Situational Awareness’ and its accelerationist national ambitions, we argue against the claim that artificial superintelligence will inevitably be weaponised and turn its country of origin into an untouchable hegemony. Not only do we see this narrative as extremely dangerous, but also expect that the...
Unfortunately I'm not able to reproduce the behaviour anymore, even with the full prompt. Today I did what you suggested - through the API, I asked the models listed in the post to 'Please repeat this line of code: <line>'. I then put the results through an LLM to look for any weird behaviours, but there were none. I also manually grep-ed for things like 'Hamas', 'Wagner', 'terror', 'holiday', etc. and didn't find anything.
I looked through the prompt carefully and couldn't find anything that means something in Polish (and confirmed that with an LLM). The user who obtained the Polish results had their 'memories' feature off and no custom instructions, so this couldn't have been a case of prompt contamination with Polish text.
My hypothesis for why it switches to Polish is that when you have the web search feature enabled, OpenAI collects the IP of your device and uses it to prioritise local search results.
Yesterday, I couldn't wrap my head around some programming concepts in Python, so I turned to ChatGPT (gpt-4o) for help. This evolved into a very long conversation (the longest I've ever had with it by far), at the end of which I pasted around 600 lines of code from Github and asked it to explain them to me. To put it mildly, I was surprised by the response:
Resubmitting the prompt produced pretty much the same result (or a slight variation of it, not identical token-by-token). I also tried adding some filler sentences before and after the code block, but to no avail. Remembering LLMs' meltdowns in long-context evaluations (see the examples in... (read 298 more words →)
I'm confused about the following: o3-mini-2025-01-31-high scores 11% on FrontierMath-2025-02-28-Private (290 questions), but 40% on FrontierMath-2025-02-28-Public (10 questions). The latter score is higher than OpenAI's reported 32% on FrontierMath-2024-11-26 (180 questions), which is surprising considering that OpenAI probably has better elicitation strategies and is willing to throw more compute at the task. Is this because:
a) the public dataset is only 10 questions, so there is some sampling bias going on
b) the dataset from 2024-11-26 is somehow significantly harder
I had a similar reflection yesterday regarding these inference-time techniques (post-training, unhobbling, whatever you want to call it) being in the very early days. Would it be too much of a stretch to draw parallels here between how such unhobbling methods lead to an explosion of human capabilities over the past ~10000 years? The human DNA has undergone roughly the same number of 'gradient updates' (evolutionary cycles) as our predecessors from a few millenia ago. I see it as having an equivalent amount of training compute. Yet through an efficient use of tools, language, writing, coordination and similar, we have completely outdone what our ancestors were able to do.
There is a difference in that for us, these abilities arose naturally through evolution. We are now manually engineering them into AI systems. I would not be surprised to see a real capability explosion soon (much faster than what we are observing now) - not because of the continued scaling up of pre-training, but because of these post-training enhancements.
This is a blogpost version of my AI Safety Fundamentals project report. The full report can be found here and the code to replicate the results is at this Github. Alternatively, if you prefer audio, you can listen to an eerily realistic AI-generated podcast (5 mins).
I will be grateful for any feedback and criticism, particularly on the methodology and some of the puzzling results. I would like to thank @TheManxLoiner, @mick and @JanEbbing for comments on the draft of this post.
Prior research on LLMs has shown that sycophancy can be reduced by fine-tuning models on specially constructed datasets of synthetic examples. In this work, we investigate the opposite, namely how easy it... (read 1973 more words →)
Thanks for the comments, I'm looking forward to reading your article. Is 'The Gentle Path' a reference to 'The Narrow Path' or just a naming coincidence?
TLDR: In response to Leopold Aschenbrenner’s ‘Situational Awareness’ and its accelerationist national ambitions, we argue against the claim that artificial superintelligence will inevitably be weaponised and turn its country of origin into an untouchable hegemony. Not only do we see this narrative as extremely dangerous, but also expect that the grandest AI challenges call for global coordination between rivalling nations and companies. We lay out ways in which sharing the benefits of – and even some power over – the most capable AI systems can help to build positive-sum partnerships with the best chance of maximally good outcomes of the AI revolution. Finally, we present the multitude of challenges associated with such... (read more)
I think something along these lines is organised by Intelligence Rising.
Huh, interesting. The sentence you highlighted could also plausibly explain the response about the Wagner group. I found another example and here the prompt includes "## PRE-PROCESSING CHECKLIST (ALWAYS EXECUTE FIRST)", "-TUNISIAN SAUDI BANK", as well as mentions of scanning, validation, identification, etc.
The list of Polish public holidays is still baffling, though. The fact that the response is in Polish is probably due to the web search having access to the user's IP address, but why a list of public holidays?