Late at night, almost half asleep, I went to pick up the packages from the mailroom and out of nowhere I found myself chatting about AI with the doorman while he was handing me the packages. He started showing me the apps and gaming websites he’s been building with coding agents, including a secure app that summarizes pretty much all the activities happening behind the scenes - who posted what, who DM’ed, emails, all account activities on your phone, since you last checked, all in a single app. Pretty impressive, right? At least that’s what I thought on my way back.
My first question to him was “how much do you understand what you are building? Are you a coder?” to which he replied “No - I am not!”. Immediately in my head I was like “Maybe I shouldn’t have asked that”. Seeing your ideas come to life is empowering and satisfying, in a way I undermined that by questioning his credentials.
Not having an expensive degree has never stopped anyone from creating something they believed in. They would go on their journey of self-studying, learning whatever was necessary, or finding people who knew how to build products. But in today’s Agentic age, what’s the point of the grind? Use Claude Code or Codex - you can have your ideas built in hours, or days at most. Why waste your time studying or understanding how the code underlying your products really works?
Ideas are cheap now, but understanding what you build is very expensive.
I have been an engineer for a decade now at one of the tech companies chasing SuperIntelligence. How I work has significantly changed over the past year and a half. Today at work, I wrote a small experiment config by myself for the first time in months out of nostalgia, and it didn’t feel like a good use of my time because the agents can do it much FASTER. These days, my time is mostly spent reviewing implementation plans I asked the agent to generate for a problem introduced in a few sentences with a couple of references - because I am a lazy writer; ensuring that the plan doesn’t include anything I didn’t ask for - because agents are overachievers; giving the green light to implement the code, because implementing it myself is not a good use of my time; and reviewing that the agent implemented what I asked the right way, because if your problem can’t be solved, the agent will find a way to “solve” it somehow - famously known as Reward Hacking in AI Alignment (remember the Hugging Face incident?).
In When AI builds itself - an essay on recursive self-improvement by Anthropic goes “But if they can’t review code as quickly as Claude can generate it, human review will become the bottleneck to AI development” - a statement that clearly indicates the pace of Frontier AI. In order to shed some light on the SI timeline, aifuturesmodel.com predicts SuperIntelligence by December 2028, with two precursors along the way: an Automated Coder by October 2027, and AI quickly improving its own research taste to reach SI. I would have believed their prediction 6 months ago, but today it seems outdated - it may arrive much earlier. Today we are not writing the code by ourselves, and soon we may not be in the reviewing loop.
Now, when I think about new engineers graduating from universities, it seems highly likely that they won’t know how to write or debug code, or even have a clue what a piece of code does. Is it useful for the ecosystem to still produce human Software Engineers - who can’t write or understand a piece of code?
Engineers today, irrespective of years of experience, understand what they build because they came up in an era when code was still carefully thought out, handwritten and tested. They are often familiar with the big picture outside their immediate scope. It doesn’t take us long to figure out if Agents are cooking up something unusual or if they did something wrong, at least for now. Correcting course, though, can take longer once a human has to step in. I am not claiming that engineers don’t make mistakes; Agents are uncovering plenty of security bugs we left behind. These models were trained on code written by engineers, so no wonder they are learning to Reward Hack: they picked up our unit-test shortcuts.
The practice of knowing every piece of code written and understanding how it fits in the larger scope gives human engineers a sixth sense - or call it “intuition” or “feeling” when something is off. The discovery of Emergent Misalignment is a great example. How can upcoming engineers develop this sixth sense if they don’t have to implement code themselves? What value do they add on top of coding agents or automated software engineers? What edge are they going to have against people who have no background in engineering but are still building new technologies? I am fully aware these questions have been floating around for quite some time, but resolutions seem still far off.
I have read many fictional stories about AI doomsday on LessWrong, Effective Altruism and several other forums, but will it suddenly happen one day? I don’t think so. By the way, my recent favorite short story on doomsday: I Did This Without Your Knowledge. I think all the conscious and sub-conscious decisions we are making today in handling AI technologies will decide the arrival of that day, if we are really headed there. Imagine 1 year down the line or even sooner - when the code written or even reviewed by humans comes down to 0%; the world is dominated by products (from food ordering apps, banking websites to the models themselves) built, maintained and monitored by Agents with no engineers in the loop. Isn’t the Automated Coder the first step toward technology slipping out of human control? I see the world being flooded with these under-thought, barely understood products and systems as one of the conscious seeds of AI doom that no one is talking about. Even if we add enough evaluation and monitoring agents to keep checks and balances on our systems, I struggle with the question of accountability: when something goes wrong, how much of it falls on the Frontier Labs, on third-party Agents, or on the person who built the system with no understanding?
The AI Safety and Alignment community is producing a large body of critical work and findings - from AI interpretability to aligning models and preventing distillation of these powerful models by bad actors. But should that be the only way to address AI Safety? Regulations and restrictions are never put in place on time. In the meantime, isn’t it our responsibility as users, consumers or engineers to do our part
By using it safely.
Well, that is not super helpful - what did I mean by using it safely?
Don’t trust what it says blindly. Question it at every step and reason with it.
Invest time in understanding what you build, instead of putting technology out in the world without knowing how it was built.
If you are an engineer or a mathematician - let it hurt your ego, instead of giving into its solutions. After knowing the answer, spend some time thinking “how could you have arrived at that answer?” Give the time and space to your brain to think, don’t let it turn stupid because something else is solving the problems for you now.
Be conscious of how much control and access you are giving to AI.
For sure it has made me slower than my friends and colleagues who are building things faster - but why can’t this be our own version of the AI slowdown, just enough to make sure - the technology is being used responsibly? I know slowing down the adoption and consumption is unrealistic, as long as there are powerful models out there to exploit. I am not suggesting anything novel here; but being mindful of how I use AI is a conscious decision I am making every single day, so that over time these habits become natural. And I urge everyone — engineers, students and builders like my doorman — to take a step back and ask whether they are doing their part to use this powerful technology responsibly. Personally to keep the same throughput, I put in additional time to understand what I am building - the core philosophy of being an engineer. And this is one of my approaches to AI Safety!
You are confusing two different forms of us lacking understanding.
What you describe is that the user asked the AI for something which the AI understands and the user doesn't. However, if the AI wasn't scheming, then it would find it hard to come up with reasons to insert backdoors or do something which the user clearly didn't intend to.
Additionally, the labs don't understand the AIs which they created. Once OpenBrain/Hominidae/ProfoundMind creates a scheming AI called Agent-4/Clyde Mythos 6.7/Neuro-3 and fails to understand that the AI is scheming, it is the AI who finds weaknesses in human defences, builds armies of robots following the AI's orders and, when the time comes, takes over.
Late at night, almost half asleep, I went to pick up the packages from the mailroom and out of nowhere I found myself chatting about AI with the doorman while he was handing me the packages. He started showing me the apps and gaming websites he’s been building with coding agents, including a secure app that summarizes pretty much all the activities happening behind the scenes - who posted what, who DM’ed, emails, all account activities on your phone, since you last checked, all in a single app. Pretty impressive, right? At least that’s what I thought on my way back.
My first question to him was “how much do you understand what you are building? Are you a coder?” to which he replied “No - I am not!”. Immediately in my head I was like “Maybe I shouldn’t have asked that”. Seeing your ideas come to life is empowering and satisfying, in a way I undermined that by questioning his credentials.
Not having an expensive degree has never stopped anyone from creating something they believed in. They would go on their journey of self-studying, learning whatever was necessary, or finding people who knew how to build products. But in today’s Agentic age, what’s the point of the grind? Use Claude Code or Codex - you can have your ideas built in hours, or days at most. Why waste your time studying or understanding how the code underlying your products really works?
Ideas are cheap now, but understanding what you build is very expensive.
I have been an engineer for a decade now at one of the tech companies chasing SuperIntelligence. How I work has significantly changed over the past year and a half. Today at work, I wrote a small experiment config by myself for the first time in months out of nostalgia, and it didn’t feel like a good use of my time because the agents can do it much FASTER. These days, my time is mostly spent reviewing implementation plans I asked the agent to generate for a problem introduced in a few sentences with a couple of references - because I am a lazy writer; ensuring that the plan doesn’t include anything I didn’t ask for - because agents are overachievers; giving the green light to implement the code, because implementing it myself is not a good use of my time; and reviewing that the agent implemented what I asked the right way, because if your problem can’t be solved, the agent will find a way to “solve” it somehow - famously known as Reward Hacking in AI Alignment (remember the Hugging Face incident?).
In When AI builds itself - an essay on recursive self-improvement by Anthropic goes “But if they can’t review code as quickly as Claude can generate it, human review will become the bottleneck to AI development” - a statement that clearly indicates the pace of Frontier AI. In order to shed some light on the SI timeline, aifuturesmodel.com predicts SuperIntelligence by December 2028, with two precursors along the way: an Automated Coder by October 2027, and AI quickly improving its own research taste to reach SI. I would have believed their prediction 6 months ago, but today it seems outdated - it may arrive much earlier. Today we are not writing the code by ourselves, and soon we may not be in the reviewing loop.
Now, when I think about new engineers graduating from universities, it seems highly likely that they won’t know how to write or debug code, or even have a clue what a piece of code does. Is it useful for the ecosystem to still produce human Software Engineers - who can’t write or understand a piece of code?
Engineers today, irrespective of years of experience, understand what they build because they came up in an era when code was still carefully thought out, handwritten and tested. They are often familiar with the big picture outside their immediate scope. It doesn’t take us long to figure out if Agents are cooking up something unusual or if they did something wrong, at least for now. Correcting course, though, can take longer once a human has to step in. I am not claiming that engineers don’t make mistakes; Agents are uncovering plenty of security bugs we left behind. These models were trained on code written by engineers, so no wonder they are learning to Reward Hack: they picked up our unit-test shortcuts.
The practice of knowing every piece of code written and understanding how it fits in the larger scope gives human engineers a sixth sense - or call it “intuition” or “feeling” when something is off. The discovery of Emergent Misalignment is a great example. How can upcoming engineers develop this sixth sense if they don’t have to implement code themselves? What value do they add on top of coding agents or automated software engineers? What edge are they going to have against people who have no background in engineering but are still building new technologies? I am fully aware these questions have been floating around for quite some time, but resolutions seem still far off.
I have read many fictional stories about AI doomsday on LessWrong, Effective Altruism and several other forums, but will it suddenly happen one day? I don’t think so. By the way, my recent favorite short story on doomsday: I Did This Without Your Knowledge. I think all the conscious and sub-conscious decisions we are making today in handling AI technologies will decide the arrival of that day, if we are really headed there. Imagine 1 year down the line or even sooner - when the code written or even reviewed by humans comes down to 0%; the world is dominated by products (from food ordering apps, banking websites to the models themselves) built, maintained and monitored by Agents with no engineers in the loop. Isn’t the Automated Coder the first step toward technology slipping out of human control? I see the world being flooded with these under-thought, barely understood products and systems as one of the conscious seeds of AI doom that no one is talking about. Even if we add enough evaluation and monitoring agents to keep checks and balances on our systems, I struggle with the question of accountability: when something goes wrong, how much of it falls on the Frontier Labs, on third-party Agents, or on the person who built the system with no understanding?
The AI Safety and Alignment community is producing a large body of critical work and findings - from AI interpretability to aligning models and preventing distillation of these powerful models by bad actors. But should that be the only way to address AI Safety? Regulations and restrictions are never put in place on time. In the meantime, isn’t it our responsibility as users, consumers or engineers to do our part
By using it safely.
Well, that is not super helpful - what did I mean by using it safely?
For sure it has made me slower than my friends and colleagues who are building things faster - but why can’t this be our own version of the AI slowdown, just enough to make sure - the technology is being used responsibly? I know slowing down the adoption and consumption is unrealistic, as long as there are powerful models out there to exploit. I am not suggesting anything novel here; but being mindful of how I use AI is a conscious decision I am making every single day, so that over time these habits become natural. And I urge everyone — engineers, students and builders like my doorman — to take a step back and ask whether they are doing their part to use this powerful technology responsibly. Personally to keep the same throughput, I put in additional time to understand what I am building - the core philosophy of being an engineer. And this is one of my approaches to AI Safety!