Introduction

📎 A superintelligence is a hypothetical agent that possesses intelligence surpassing that of the brightest and most gifted human minds. "Superintelligence" may also refer to a property of advanced problem-solving systems that excel in specific areas (e.g., superintelligent language translators or engineering assistants). Nevertheless, a general-purpose superintelligence remains hypothetical, and its creation may or may not be triggered by an intelligence explosion or a technological singularity. [1]

After the large model research explosion in the first half of 2025, we now possess numerous Large Reasoning Models (LRM) capable of solving various complex problems. These models have learned to reflect and communicate in anthropomorphic tones through large-scale reinforcement learning training [2] . Additionally, more powerful large model-based agents exhibiting enhanced capabilities in both planning and tool utilization have emerged, such as computer user agents [3] that can control our computers to help us manage travel plans and bookings. It appears that we are really in the early stages of what is referred to as superintelligence, where AI would far surpass human cognitive abilities.

Over the past 2-3 years, a recurring pattern has emerged in large model research: whenever a new class of models is released, a wave of researchers quickly shifts focus to explore modality and task transfer, risk assessment, and interpretability. Meanwhile, others work on developing models that surpass the current state-of-the-art. However, research aimed at actually mitigating model risks remains noticeably scarce: often, just as we identify a vulnerability and begin to explore solutions, a newer, larger-scale model, built on a more advanced paradigm with even better performance is already released😭. As written in OpenAI’s Superalignment team introduction in 2023, “We need scientific and technical breakthroughs to steer and control AI systems much smarter than us.” But as we continue to witness new AI systems exhibiting sycophantic behavior, ****specification gaming, and other concerning traits [5] , we must seriously consider the possibility that one day, regulation and control may simply become infeasible.

At times, I even wonder whether it's the models that are now training us: th

...

Posts

Wikitag Contributions

Comments