Researchers are more worried than most of the population about the rapid pace at which frontier AI models are developing. This is because the paradigms that are supposed to be the basis for preventing and defending against possible harms are still under study. Regulation of AI governance has been proposed by multiple researchers, not only belonging to the technical field but also those working in ethics and policy. However, there is still minimal discussion happening around scale and impact of AGI, (and in the future, ASI) by prospective regulatory bodies (like governments) in developing defence mechanisms, or drawing a baseline of requirements to be satisfied by vastly or selectively deployed frontier AI models. Everyone wants to talk about building such a system; but only few want to discuss the repercussions.
Currently, you will come across two groups of people. One making the case that AI is just a normal technology, and it will follow the path of existing technologies, dissolve and dilute into our lives as a part of our being. Others argue that proceeding at an unprecedented speed, we might have AGI (artificial general intelligence) by 2027, and ASI(artificial superintelligence) by 2030, transforming the world. Both perspectives hold some truth to them. I recently completed the AGI strategy course provided by BlueDot Impact, a non-profit organisation working towards beneficial AI. In this article, I break down everything I learnt from the curriculum and live discussions on AI Safety, from an infrastructural (non technical) perspective.
AGI VS ASI - What’s the deal?
AGI stands for Artificial General Intelligence. It is essentially a type of AI that can match or surpass (to an extent) human capabilities in performing cognitive tasks, such as self-teaching/self-awareness, adapting to situations without specific reprogramming, etc. This is next stage after the current one, ANI (Artificial Narrow Intelligence), where the models remain in a restricted environment and select parameters. Contrary to popular belief, even highly advanced LLMs by even the top tech companies are not visibly at the AGI stage yet, though some models are said to show signs of self-awareness, without intention. AGI is a competitive topic of research in the race right now.
ASI stands for Artificial SuperIntelligence. This is the not-so-far-fetched step after AGI, where a machine’s cognitive capabilities completely surpass the human mind. It will have the ability to constantly rewrite and improve its code, operating at exponentially higher intelligence than humans in almost all fields. Researchers predict that once AGI is achieved, ASI will come knocking on the door, as the commercialisation of big tech companies won’t allow them to fall behind each other.
The Triad, Scaling and the drivers of progress
Before beginning, let’s have a look at some important definitions:
FLOP: a single basic mathematical operation (like addition or multiplication) that a computer performs. The total number of FLOP is a measure of computational work done during AI training.
Compute efficiency is a measure of how “good” an AI model you get with a given financial investment.
Hardware price-performance measures the amount of computational resources (FLOP) available for a given financial investment.
Algorithmic efficiency measures how “good” of an AI model you get with a given number of FLOP.
Ethan Mollick, in his article (link), explains scaling in simple terms as, “larger models require more computing power, measured in FLOPs, and generally seem to be “smarter” overall.” This is the easiest way to explain how scaling is a huge factor in driving AI progress. The largest models available at any given time are called the frontier models. On that basis, the author classifies the models into 4 generations, and an extended fifth one, comprehensively discussed in the aforementioned article. My purpose of discussing this is the new form of scale explored, “thinking”. Amazing levels of performances are now being achieved by models by performing multiple internal reasoning steps before producing an output, which can lead to more accurate responses, which is termed as thinking. This happens AFTER a model is trained, and has a scaling law of its own. Since the time when this article was published, this alternate scaling approach has already been adopted by the current highest gen models, but it was first implemented by OpenAI. Elaborating on the chain-of-thought reasoning, which prompts the model to look up the data and choose the best one from multiple options, forcing it to “think” in steps, what OpenAI did was get the o1 models to go through just this sort of “thinking” process, producing hidden thinking tokens before giving a final answer. In doing so, they revealed another scaling law : the longer a model “thinks,” the better its answer is.
Pathways to harm, Defence Mechanisms & The (Mis)Alignment Problem
A. Pathways to harm
Identifying, analyzing and prioritizing the possible threat pathways that AGI is liable to accompany is the first step in the attempt to build safe AI systems. The following is an objective overview of the catastrophic harm that could be caused if systems go beyond control:
Power concentration: Imagine this. A small group gains control of frontier AI systems and uses them to manipulate elections through micro-targeted disinformation, fake evidence, and synthetic media so convincing that truth becomes unknowable. They deploy AI-powered surveillance that tracks every digital interaction, predicts dissent before it emerges, and automatically suppresses opposition through economic and social levers. An example already exists: the infamous Meta Ray-Ban glasses. Within months, autonomous weapons patrol the streets, facial recognition determines access to resources, and AI advisors whisper into the ears of every official - all controlled by those who seized the models first. Democratic institutions become theatrical performances while real power flows through algorithmic systems accountable to no one. Whether it’s a government using AI to become permanently unremovable, a corporation achieving regulatory capture through superintelligent lobbying, or an AI system itself accumulating power while maintaining a facade of servitude, the result is the same: the permanent concentration of control in hands that will never voluntarily let go.
Gradual Disempowerment: As AI systems become cheaper and more capable than humans at nearly every task, companies face overwhelming pressure to automate. First it’s customer service and data entry, then management and creative work, eventually even governance and resource allocation. This could lead to humans losing economic leverage as their labour becomes worthless. Our ability to influence culture might degrade as AI generates all the content we consume. Our political power could dissipate as governments pivot to AI for their tax revenue, administration, and security, instead of human income tax, human civil servants, and human military personnel. We might wake up one day to find ourselves comfortable but irrelevant - living in a world we no longer understand or control, where the economy optimises for AI activities rather than human needs, and our preferences matter as little as those of retired plough horses.
Catastrophic pandemics: Advanced AI could dramatically lower the barriers to creating pandemic pathogens. Today, designing a novel virus requires rare expertise, expensive equipment, and years of work. In a few years, AI could walk someone through synthesising a pathogen more transmissible than measles and deadlier than rabies. Unlike natural pandemics that evolve toward transmission over lethality, an engineered pathogen could be optimised for maximum harm - combining long asymptomatic periods with high mortality, resistance to existing treatments, or even targeting specific genetic markers. The same AI capabilities that accelerate beneficial biomedical research - protein folding prediction, drug discovery, genetic engineering - could be misused by state bioweapons programmes, terrorist groups, or disturbed individuals to create humanity’s final pandemic.
Critical infrastructural collapse: Modern infrastructure runs on outdated industrial control systems and unpatched software that were never designed for internet connectivity. AI could systematically map these vulnerabilities across utilities, hospitals, and supply chains, then coordinate attacks that would take human hackers months to plan and execute. AI-enabled attackers could compromise the industrial control systems that run power plants, the networks that manage water treatment, and the logistics software that routes food and medicine - not enough to collapse civilisation, but enough to cause sustained disruptions that compound over weeks. When the power grid has rolling blackouts, water pressure drops, hospitals run on backup generators, and just-in-time supply chains fail, even a wealthy nation starts to fray at the edges.
B. Defence Mechanisms & the (mis)alignment problem
There are three layers of defence to tackle the aforementioned threats:
Prevent dangerous AI training: This is unarguably the first layer of defence. Interventions that (mostly) live here: export controls and the most advanced AI chips, international agreements and regulations not to train AI systems that cross certain red lines, “safe by design” AI systems, and norms in the research community not to race ahead hurriedly. Prevention feels the cleanest. If nothing dangerous gets built, nothing bad can happen. But it’s also one of the hardest layers to establish. There’s intense commercial competition between AI companies, and intense rivalry between nations. Most actors are incentivised to race ahead. Some of the proposed solutions (from a governance perspective) that envision the implementation of this layer essentially enlist the following steps: Infrastructurally: -An independent authority overseeing frontier AI development, holding government initiatives accountable, and licensing and auditing private labs. This would need real enforcement power with fines, license revocation etc. -National regulators need to align on shared safety standards, evaluation benchmarks, and incident-reporting through a centralized body or a network of agreements. -Non-commercial labs (govt funded) focused purely on alignment and safety research, insulated from competitive pressure, so that they don’t have to be a part of the “race” and actual study could be conducted -laws that ensure the protection of human labour from automation. Refer to these for a more comprehensive view of the proposals: The Project : https://situational-awareness.ai/the-project/?utm_source=bluedot-impact International AI Treaty: https://aitreaty.org/?utm_source=bluedot-impact
Constrain dangerous AI capabilities: When AI companies train their next generation of AI systems, they don’t know what that AI system will be capable of. They know that if they make it bigger and train it for longer, it will be better, but they don’t know how much better and in what way. This also means that they don’t know how dangerous their next generation of AI systems will be. They discover that during and after the training process. Then they implement safeguards to try to prevent their AI systems from taking harmful actions. So, the sole way to practice this (and the next layer) is to actually understand what is happening inside the model, because “intent” cannot be quantified. So even if regulations are implemented by a body, the ultimate test is whether the model aligns with the bare minimum threshold set proactively or not. And in case misalignment and misuse occur, how will they be controlled? This is the gap that stands as a barrier between developing frontier AI models and developing “safe” , “aligned” and “controlled” AI systems, along with ensuring that they remain isolated, or can withstand bad actors. (this could be humans, or models themselves.) Interpretability researchers are hard at work with alignment researchers because the combination of both will give birth to these systems. To read more on alignment, and the difference between alignment, misalignment and AI control, refer to these resources: What is AI alignment? : link AI Control: link
Withstand dangerous AI actions: If dangerous AI models roam wild, we need to harden society against a possible enslaught of attacks. Imagine what it would look like to have a pandemic-proof, where no matter how good an AI system is at building bioweapons, we detect and contain pathogens immediately. Imagine what it would look like for critical national infrastructure to be secure against physical and cyber attacks. We might have the most capable attackers working “on our side”, testing the defences, highlighting where there are weaknesses, and using their expertise to shore up the defences. This is the layer that considers the worst-case scenario. If we encounter systematic risks, cyberattacks, Rogue AI (deceptive, self-aware), misaligned AI, are we capable enough to tackle these? If the first two layers fail and control is lost, this final layer needs to be stronger than steel (metaphorically) to prevent diverse atrocious effects on human civilisation.
From my understanding of all the discussions and the material provided, I gathered that the major discrepancies are as follows:
The lack of awareness and action in the direction of planning for the defensive layers on a larger and steadfast scale
The need to baseline “intent”
Thus I think the first and foremost step to establish any defensive layer is inevitably increasing the research work in progress, to answer the questions raised by misalignment etc., and to decide a threshold of values. If nobody acts on these, then if the threats become out of control, it might be too late to undo the effects of it. Defence should be our form of attack. So the intervention plan relies heavily on the progress of the fields of AI alignment, AI control and Interpretability. Interpretability, I believe, could be the biggest tool in hand to understand behaviour, jailbreaking (re: cyber/bio/terrorist attacks), and even emotion recognition. The faster we can recognise and detect these, the better we can handle their effects in case of loss of control. This should give us a strategic advantage.
This work connects to the larger safety landscape by feeding into the technical groundwork needed for any credible regulatory or institutional defence layer. You can’t set meaningful safety thresholds or govern frontier AI without first being able to see what’s actually happening inside these models.
This rests on a few shaky assumptions: that interpretability research can keep pace with capability growth rather than perpetually trailing it; that its insights are usable by labs/regulators in real time, not just academically interesting; and that “intent” can eventually be pinned down as something measurable rather than staying philosophically fuzzy.
While that remains for the policymakers to decide, I will continue my research in the technical aspect of this quotient. Stay tuned, and thanks for reading!
Researchers are more worried than most of the population about the rapid pace at which frontier AI models are developing. This is because the paradigms that are supposed to be the basis for preventing and defending against possible harms are still under study. Regulation of AI governance has been proposed by multiple researchers, not only belonging to the technical field but also those working in ethics and policy. However, there is still minimal discussion happening around scale and impact of AGI, (and in the future, ASI) by prospective regulatory bodies (like governments) in developing defence mechanisms, or drawing a baseline of requirements to be satisfied by vastly or selectively deployed frontier AI models. Everyone wants to talk about building such a system; but only few want to discuss the repercussions.
Currently, you will come across two groups of people. One making the case that AI is just a normal technology, and it will follow the path of existing technologies, dissolve and dilute into our lives as a part of our being. Others argue that proceeding at an unprecedented speed, we might have AGI (artificial general intelligence) by 2027, and ASI(artificial superintelligence) by 2030, transforming the world. Both perspectives hold some truth to them. I recently completed the AGI strategy course provided by BlueDot Impact, a non-profit organisation working towards beneficial AI. In this article, I break down everything I learnt from the curriculum and live discussions on AI Safety, from an infrastructural (non technical) perspective.
AGI VS ASI - What’s the deal?
AGI stands for Artificial General Intelligence. It is essentially a type of AI that can match or surpass (to an extent) human capabilities in performing cognitive tasks, such as self-teaching/self-awareness, adapting to situations without specific reprogramming, etc. This is next stage after the current one, ANI (Artificial Narrow Intelligence), where the models remain in a restricted environment and select parameters. Contrary to popular belief, even highly advanced LLMs by even the top tech companies are not visibly at the AGI stage yet, though some models are said to show signs of self-awareness, without intention. AGI is a competitive topic of research in the race right now.
ASI stands for Artificial SuperIntelligence. This is the not-so-far-fetched step after AGI, where a machine’s cognitive capabilities completely surpass the human mind. It will have the ability to constantly rewrite and improve its code, operating at exponentially higher intelligence than humans in almost all fields. Researchers predict that once AGI is achieved, ASI will come knocking on the door, as the commercialisation of big tech companies won’t allow them to fall behind each other.
The Triad, Scaling and the drivers of progress
Before beginning, let’s have a look at some important definitions:
Ethan Mollick, in his article (link), explains scaling in simple terms as, “larger models require more computing power, measured in FLOPs, and generally seem to be “smarter” overall.” This is the easiest way to explain how scaling is a huge factor in driving AI progress. The largest models available at any given time are called the frontier models. On that basis, the author classifies the models into 4 generations, and an extended fifth one, comprehensively discussed in the aforementioned article. My purpose of discussing this is the new form of scale explored, “thinking”. Amazing levels of performances are now being achieved by models by performing multiple internal reasoning steps before producing an output, which can lead to more accurate responses, which is termed as thinking. This happens AFTER a model is trained, and has a scaling law of its own. Since the time when this article was published, this alternate scaling approach has already been adopted by the current highest gen models, but it was first implemented by OpenAI. Elaborating on the chain-of-thought reasoning, which prompts the model to look up the data and choose the best one from multiple options, forcing it to “think” in steps, what OpenAI did was get the o1 models to go through just this sort of “thinking” process, producing hidden thinking tokens before giving a final answer. In doing so, they revealed another scaling law : the longer a model “thinks,” the better its answer is.
Pathways to harm, Defence Mechanisms & The (Mis)Alignment Problem
A. Pathways to harm
Identifying, analyzing and prioritizing the possible threat pathways that AGI is liable to accompany is the first step in the attempt to build safe AI systems. The following is an objective overview of the catastrophic harm that could be caused if systems go beyond control:
Imagine this.
A small group gains control of frontier AI systems and uses them to manipulate elections through micro-targeted disinformation, fake evidence, and synthetic media so convincing that truth becomes unknowable. They deploy AI-powered surveillance that tracks every digital interaction, predicts dissent before it emerges, and automatically suppresses opposition through economic and social levers. An example already exists: the infamous Meta Ray-Ban glasses.
Within months, autonomous weapons patrol the streets, facial recognition determines access to resources, and AI advisors whisper into the ears of every official - all controlled by those who seized the models first. Democratic institutions become theatrical performances while real power flows through algorithmic systems accountable to no one.
Whether it’s a government using AI to become permanently unremovable, a corporation achieving regulatory capture through superintelligent lobbying, or an AI system itself accumulating power while maintaining a facade of servitude, the result is the same: the permanent concentration of control in hands that will never voluntarily let go.
As AI systems become cheaper and more capable than humans at nearly every task, companies face overwhelming pressure to automate. First it’s customer service and data entry, then management and creative work, eventually even governance and resource allocation.
This could lead to humans losing economic leverage as their labour becomes worthless. Our ability to influence culture might degrade as AI generates all the content we consume. Our political power could dissipate as governments pivot to AI for their tax revenue, administration, and security, instead of human income tax, human civil servants, and human military personnel.
We might wake up one day to find ourselves comfortable but irrelevant - living in a world we no longer understand or control, where the economy optimises for AI activities rather than human needs, and our preferences matter as little as those of retired plough horses.
Advanced AI could dramatically lower the barriers to creating pandemic pathogens. Today, designing a novel virus requires rare expertise, expensive equipment, and years of work. In a few years, AI could walk someone through synthesising a pathogen more transmissible than measles and deadlier than rabies.
Unlike natural pandemics that evolve toward transmission over lethality, an engineered pathogen could be optimised for maximum harm - combining long asymptomatic periods with high mortality, resistance to existing treatments, or even targeting specific genetic markers.
The same AI capabilities that accelerate beneficial biomedical research - protein folding prediction, drug discovery, genetic engineering - could be misused by state bioweapons programmes, terrorist groups, or disturbed individuals to create humanity’s final pandemic.
Modern infrastructure runs on outdated industrial control systems and unpatched software that were never designed for internet connectivity. AI could systematically map these vulnerabilities across utilities, hospitals, and supply chains, then coordinate attacks that would take human hackers months to plan and execute.
AI-enabled attackers could compromise the industrial control systems that run power plants, the networks that manage water treatment, and the logistics software that routes food and medicine - not enough to collapse civilisation, but enough to cause sustained disruptions that compound over weeks. When the power grid has rolling blackouts, water pressure drops, hospitals run on backup generators, and just-in-time supply chains fail, even a wealthy nation starts to fray at the edges.
B. Defence Mechanisms & the (mis)alignment problem
There are three layers of defence to tackle the aforementioned threats:
This is unarguably the first layer of defence.
Interventions that (mostly) live here: export controls and the most advanced AI chips, international agreements and regulations not to train AI systems that cross certain red lines, “safe by design” AI systems, and norms in the research community not to race ahead hurriedly.
Prevention feels the cleanest. If nothing dangerous gets built, nothing bad can happen. But it’s also one of the hardest layers to establish. There’s intense commercial competition between AI companies, and intense rivalry between nations. Most actors are incentivised to race ahead.
Some of the proposed solutions (from a governance perspective) that envision the implementation of this layer essentially enlist the following steps:
Infrastructurally:
-An independent authority overseeing frontier AI development, holding government initiatives accountable, and licensing and auditing private labs. This would need real enforcement power with fines, license revocation etc.
-National regulators need to align on shared safety standards, evaluation benchmarks, and incident-reporting through a centralized body or a network of agreements.
-Non-commercial labs (govt funded) focused purely on alignment and safety research, insulated from competitive pressure, so that they don’t have to be a part of the “race” and actual study could be conducted
-laws that ensure the protection of human labour from automation.
Refer to these for a more comprehensive view of the proposals:
The Project :
https://situational-awareness.ai/the-project/?utm_source=bluedot-impact
International AI Treaty:
https://aitreaty.org/?utm_source=bluedot-impact
When AI companies train their next generation of AI systems, they don’t know what that AI system will be capable of. They know that if they make it bigger and train it for longer, it will be better, but they don’t know how much better and in what way.
This also means that they don’t know how dangerous their next generation of AI systems will be. They discover that during and after the training process. Then they implement safeguards to try to prevent their AI systems from taking harmful actions.
So, the sole way to practice this (and the next layer) is to actually understand what is happening inside the model, because “intent” cannot be quantified. So even if regulations are implemented by a body, the ultimate test is whether the model aligns with the bare minimum threshold set proactively or not. And in case misalignment and misuse occur, how will they be controlled? This is the gap that stands as a barrier between developing frontier AI models and developing “safe” , “aligned” and “controlled” AI systems, along with ensuring that they remain isolated, or can withstand bad actors. (this could be humans, or models themselves.) Interpretability researchers are hard at work with alignment researchers because the combination of both will give birth to these systems.
To read more on alignment, and the difference between alignment, misalignment and AI control, refer to these resources:
What is AI alignment? : link
AI Control: link
If dangerous AI models roam wild, we need to harden society against a possible enslaught of attacks.
Imagine what it would look like to have a pandemic-proof, where no matter how good an AI system is at building bioweapons, we detect and contain pathogens immediately.
Imagine what it would look like for critical national infrastructure to be secure against physical and cyber attacks. We might have the most capable attackers working “on our side”, testing the defences, highlighting where there are weaknesses, and using their expertise to shore up the defences.
This is the layer that considers the worst-case scenario. If we encounter systematic risks, cyberattacks, Rogue AI (deceptive, self-aware), misaligned AI, are we capable enough to tackle these? If the first two layers fail and control is lost, this final layer needs to be stronger than steel (metaphorically) to prevent diverse atrocious effects on human civilisation.
From my understanding of all the discussions and the material provided, I gathered that the major discrepancies are as follows:
Thus I think the first and foremost step to establish any defensive layer is inevitably increasing the research work in progress, to answer the questions raised by misalignment etc., and to decide a threshold of values. If nobody acts on these, then if the threats become out of control, it might be too late to undo the effects of it. Defence should be our form of attack. So the intervention plan relies heavily on the progress of the fields of AI alignment, AI control and Interpretability. Interpretability, I believe, could be the biggest tool in hand to understand behaviour, jailbreaking (re: cyber/bio/terrorist attacks), and even emotion recognition. The faster we can recognise and detect these, the better we can handle their effects in case of loss of control. This should give us a strategic advantage.
This work connects to the larger safety landscape by feeding into the technical groundwork needed for any credible regulatory or institutional defence layer. You can’t set meaningful safety thresholds or govern frontier AI without first being able to see what’s actually happening inside these models.
This rests on a few shaky assumptions: that interpretability research can keep pace with capability growth rather than perpetually trailing it; that its insights are usable by labs/regulators in real time, not just academically interesting; and that “intent” can eventually be pinned down as something measurable rather than staying philosophically fuzzy.
While that remains for the policymakers to decide, I will continue my research in the technical aspect of this quotient. Stay tuned, and thanks for reading!