My use of “must” wasn’t just about technical necessity, but rather a philosophical or strategic imperative — that we ought to inform AGI not only through recent trends in deep learning (say, post-2014), but also by drawing from longer-standing academic traditions, like epistemic logic. 

I completely agree on the importance of strategic thinking. Personally, I like to hear what early AI pioneers had to say about modeling AI. For example, Minsky's society of mind. I believe the trend of AI must be informed by the development of epistemology, and I’ve basically bet my research on the idea that epistemological progress will shape AGI

I think LLMs are even worse — not just with rare encodings, but also when it comes to reasoning with rare structures. Theory-of-mind tasks provide good evidence for this. LLMs aren't good at inferring others’ mental states; rather, they tend to mimic reasoning when reasoning steps are present in the training data.

This is a highly intriguing research finding. It seems consistent with observations in multi-modal models, where different data types can effectively jailbreak each other.

At the same time, unlike visual reasoning, code is processed entirely in natural language. This suggests two possible approaches to analyzing the underlying cause.

1. Data Type: Analyzing the unique characteristics of coding, compared to natural language, may help explain this phenomenon.

2. Representation: Examining which neurons change during fine-tuning and analyzing their correlations could provide a clearer causal explanation.

Based on your experimental insights, which approach do you think is more effective for identifying the cause of this phenomenon?

Curious to hear your thoughts!

I'm really interested in AI and want to build something amazing, so I’m always looking to expand my imagination! Sure, research papers are full of ideas, but I feel like insights into more universal knowledge spark a different kind of creativity. I found LessWrong through things like LLM, but the posts here give me the joy of exploring a much broader world!

I’m deeply interested in the good and bad of AI. While aligning AI with human values is important, alignment can be defined in many ways. I have a bit of a goal to build up my thoughts on what’s right or wrong, what’s possible or impossible, and write about them.