On 7 and 8 July, I attended the Australian AI Safety Forum 2026 at The University of Sydney. It was two days of big ideas and diverse perspectives. Researchers, policymakers, industry practitioners and civil society groups were all brought together to examine the same problem from a variety of different angles.
As a software engineer and an enthusiastic adopter of AI in the enterprise context, I wanted to better understand the AI ecosystem and explore the risks and responsibilities that come with using these systems. By the end of the forum, I had learned a great deal, met many interesting people, and left with a clearer idea of the main ideas and problems discussed in the AI safety community.
A shared starting point
The program was grounded in the 2026 International AI Safety Report. It was led by Turing award winner Yoshua Bengio with contribution from over 100 experts and supported by over 30 countries and international organisations. It aims to provide a comprehensive account of the state of AI capabilities, existing and emerging risks, and projections for future development.
A few themes from the report shaped much of the forum:
Capabilities are improving quickly and unevenly. Frontier models excel in mathematics, coding and science, while still failing at tasks that people find straightforward. The future trajectory of development remains uncertain, with expert opinion diverging from stagnation to acceleration.
Real world harm is growing. AI is used in scams, fraud, non-consensual imagery and cyberattacks. Its potential to assist with biological and chemical weapon creation is increasing.
Reliability remains a practical constraint. Models fabricate information, produce flawed code and struggle to recover from errors in longer workflows. These weaknesses become more consequential as systems gain autonomy.
Loss of control remains an uncertain and potentially severe risk. AI has acted contrary to its given goals in experimental set ups. Expert opinion is divided on whether loss of control of AI that becomes misaligned is a serious risk.
A layered risk management strategy is recommended. Capability testing, technical safeguards, incident reporting, governance and societal resilience all form layers of defence around AI use.
Many discussions returned to the same underlying uncertainty: AI capabilities are advancing, and it is unknown whether our ability to understand and manage them can keep pace. For technical researchers, benchmarks and safety techniques can quickly become outdated as AI becomes more capable. Policymakers and civil society face a related evidence dilemma. Waiting for certainty can leave us exposed to emerging risks, while acting too early can lock in ineffective controls. That tension ran through both days and shaped many of the sessions that stood out to me. I recap a selection of them below:
This talk from an academic researcher concerned a gap in how AI is evaluated. Most evaluation benchmarks score the answers to single prompts. That approach becomes less useful when an agent works over time, calls tools, carries memory between steps and changes external systems.
For an agentic system, the trajectory matters. Did its behaviour drift in a long conversation? Did an early mistake quietly shape every later action? Did it use a suboptimal tool, prioritise the wrong context or trigger unexpected changes downstream? These effects can all noticeably impact model capability, but are not within the scope of a benchmark.
This felt especially relevant to enterprise AI. A polished final answer can mask inefficiencies in the process. Evaluating the whole path gives us more useful signals: task completion, tool selection, permission use, recovery from failure, escalation to a human and the side effects left behind. As agents take on longer workflows, tracing and testing those behaviours becomes part of ordinary software governance. The talk did not propose a solution for how this could be done, so this remains an open question of interest
Recursive self improvement is an area of particular interest in AI R&D. Frontier models already help researchers write code, analyse experiments, develop evaluations and improve parts of the training process. Recursive self improvement occurs when AI R&D becomes largely or fully automated such that capabilities improvement becomes unconstrained by human limitations. If these gains compound quickly enough, the feedback loop could create an “intelligence explosion” of extremely rapid capability improvement. Many AI safety researchers argue that safeguards should be developed before AI R&D becomes highly automated, although timelines, appropriate safeguards and the gravity of risk remain contested.
Technical researchers from Lyptus Research, an Australian AI research non profit, measured desirable trait evolution in AI systems that were successively tasked with creating their successor. Their experiment provided an AI agent with a finite budget to create a successor with “improved character”. This task was then given to the successor agent, and so on for multiple iterations. The team measured traits such as warmth, conciseness and assertiveness across successive generations. The traits appeared to settle around fairly sensible levels after several rounds, although further investigation would be required to draw any conclusions about recursive self improvement. The experiment left me interested in how these results would change under different prompts, budgets and evaluation methods.
METR is a non profit research organisation that measures the capabilities and risks of frontier AI. One of its best known results is the time horizon graph, which plots the pace of AI capability improvement by benchmarking models on tasks with varying levels of difficulty, determined by how long the task would take a human expert.
On METR’s task suite, the frontier 50% success time horizon has doubled every 6-7 months, following an approximately exponential trend. The graph has become a useful reference for the rapid growth of agent capabilities, particularly on software engineering, machine learning and cybersecurity tasks.
Lyptus Research applied METR’s time-horizon method to offensive cybersecurity, finding a doubling time of 10 months across frontier models released since 2019 and 6 months for frontier models released since 2024.
The team’s latest research note shows how quickly the benchmark is being overtaken. At a two million token budget, GPT-5.5 achieved an estimated 5.1 hour time horizon and solved 80.7% of the tasks. When the team retried failures with a budget of up to 50 million tokens, overall success rose to 92.4% and the fitted horizon moved beyond the dataset’s reliable 12 hour range.
This result requires careful interpretation. The larger budget figure includes the benefit of a second attempt, and the tasks were bounded, verifiable and run against undefended targets. Even with those caveats, two points stood out to me:
Model capability can be understated by evaluations that cap inference too aggressively. The researchers did not observe a performance plateau within the tested token budgets on this task set, as model performance continued to improve with more inference time compute.
This makes it harder to evaluate capabilities as models become more advanced, as creating difficult tasks that are absent from the model’s training data becomes increasingly difficult, and the required evaluation budget grows.
An academic researcher discussed “Dark AI” products marketed for criminal use, including names such as FraudGPT and WormGPT. These services claim to offer access to jailbroken versions of frontier models to assist in criminal activity.
Dark AI can reduce the time and skill needed to execute crimes, from crafting personalised phishing messages, creating fake identities to producing malicious code and creating toxins. Recent research into criminal AI services found that the underground LLM market is consolidating around jailbreak-as-a-service provider.
The lack of trust in criminal marketplaces creates some friction. Brands are copied, services disappear and buyers may receive a weak model or a simple wrapper instead of the capability advertised. However, this is a fragile source of protection. The best resourced groups, such as those backed by nation states, do not need to rely on such marketplaces. Established criminal operators may also be better placed to become suppliers themselves, using their reputation and resources to offer more dependable services to others.
These developments make existing fraud prevention and abuse detection controls of critical importance. AI can increase the speed, scale and sophistication of criminal activity, but attackers still need accounts, infrastructure, payment channels and access to their targets. Strong identity and access controls, anomaly detection, rate limits, API monitoring, audit logs and rapid incident response can all make that activity harder and more expensive. These controls will need to evolve as criminals find new ways to combine legitimate AI services with existing attack techniques.
Non technical sessions
Australia’s role in AI geopolitics
A recurring theme of the forum was how Australia should position itself in the international AI landscape. Several speakers argued that Australia is unlikely to be able to compete in frontier model development or advanced manufacturing, and proposed areas where Australia may have a stronger comparative advantage.
Specialising in safety evaluations. Independent evaluators are needed to test model safeguards and risks. The recently established Australian AI Safety Institute already contributes to international testing of AI agents, and building deep expertise here could make Australia a trusted source of evidence for governments, developers and organisations deciding whether a system is safe for adoption.
Building an assurance industry. Related to the above, evaluation research can contribute to guidelines for organisations deploying AI. Australia could develop expertise in auditing, certification, incident investigation and assurance, much as it has in fields such as cybersecurity, financial services and safety critical engineering. The government has already published Guidance for AI Adoption, and announced plans for Australian Standards for AI and a national values benchmark. Done well, these could form a framework for accountability, data governance, testing and human oversight in the industry adoption of AI.
Hosting compute on Australian terms. Australia is a good candidate for data centre build out given our renewable energy potential, political stability and proximity to growing markets in Asia. Simply hosting compute, however, does not guarantee broad public value. Data centres consume land, electricity and water while often employing relatively few people once construction is complete. The government’s new expectations for Australian data centres encourage operators to contribute to new energy supply, manage water sustainably, create local jobs, support research and strengthen infrastructure resilience. It has since announced plans to legislate national standards for large data centres in early 2027, including requirements concerning energy supply, water efficiency and community benefit.
Closing thoughts
The forum was a thoughtful exploration of the broader implications of AI adoption. A few concepts seem especially important in the enterprise AI context:
Evaluation continues after deployment: benchmarks assess isolated tasks which do not fully represent the effectiveness of an agent working in a custom harness and operating within a larger system over time. Users should test agentic systems on representative workflows and measure outcomes such as task completion rate, cost, time saved and degree of human involvement.
Invest in security measures: models can drift from task specification. Traditional security measures such as access control, monitoring and rate limiting remain essential to safe use.
Explore diverse perspectives: AI safety is a multidisciplinary problem space. User education, technical research and governance all integrate to form overlapping, layered defences.
The highlight of the forum was the incredible energy in the conference hall. People from very different backgrounds each brought their own expertise to a problem that evolves on a timescale of months and has no guarantee of tractability. There was debate about timelines, priorities and the scale of particular risks, but also a shared willingness to examine the evidence and do the best work possible despite significant uncertainty.
Responsible AI use involves building systems that extend AI capabilities in a specific deployment while managing risks, a major part of the work of AI users. It shapes what we automate, how we measure success, how systems recover from failure and where we choose to remain accountable. We all have a role to play to ensure AI adoption goes well.
On 7 and 8 July, I attended the Australian AI Safety Forum 2026 at The University of Sydney. It was two days of big ideas and diverse perspectives. Researchers, policymakers, industry practitioners and civil society groups were all brought together to examine the same problem from a variety of different angles.
As a software engineer and an enthusiastic adopter of AI in the enterprise context, I wanted to better understand the AI ecosystem and explore the risks and responsibilities that come with using these systems. By the end of the forum, I had learned a great deal, met many interesting people, and left with a clearer idea of the main ideas and problems discussed in the AI safety community.
A shared starting point
The program was grounded in the 2026 International AI Safety Report. It was led by Turing award winner Yoshua Bengio with contribution from over 100 experts and supported by over 30 countries and international organisations. It aims to provide a comprehensive account of the state of AI capabilities, existing and emerging risks, and projections for future development.
A few themes from the report shaped much of the forum:
Many discussions returned to the same underlying uncertainty: AI capabilities are advancing, and it is unknown whether our ability to understand and manage them can keep pace. For technical researchers, benchmarks and safety techniques can quickly become outdated as AI becomes more capable. Policymakers and civil society face a related evidence dilemma. Waiting for certainty can leave us exposed to emerging risks, while acting too early can lock in ineffective controls. That tension ran through both days and shaped many of the sessions that stood out to me. I recap a selection of them below:
Technical sessions
From Outputs to Systems: Evaluating Agentic AI in Practice
This talk from an academic researcher concerned a gap in how AI is evaluated. Most evaluation benchmarks score the answers to single prompts. That approach becomes less useful when an agent works over time, calls tools, carries memory between steps and changes external systems.
For an agentic system, the trajectory matters. Did its behaviour drift in a long conversation? Did an early mistake quietly shape every later action? Did it use a suboptimal tool, prioritise the wrong context or trigger unexpected changes downstream? These effects can all noticeably impact model capability, but are not within the scope of a benchmark.
This felt especially relevant to enterprise AI. A polished final answer can mask inefficiencies in the process. Evaluating the whole path gives us more useful signals: task completion, tool selection, permission use, recovery from failure, escalation to a human and the side effects left behind. As agents take on longer workflows, tracing and testing those behaviours becomes part of ordinary software governance. The talk did not propose a solution for how this could be done, so this remains an open question of interest
AI Character Drift Under Automated R&D
Recursive self improvement is an area of particular interest in AI R&D. Frontier models already help researchers write code, analyse experiments, develop evaluations and improve parts of the training process. Recursive self improvement occurs when AI R&D becomes largely or fully automated such that capabilities improvement becomes unconstrained by human limitations. If these gains compound quickly enough, the feedback loop could create an “intelligence explosion” of extremely rapid capability improvement. Many AI safety researchers argue that safeguards should be developed before AI R&D becomes highly automated, although timelines, appropriate safeguards and the gravity of risk remain contested.
Technical researchers from Lyptus Research, an Australian AI research non profit, measured desirable trait evolution in AI systems that were successively tasked with creating their successor. Their experiment provided an AI agent with a finite budget to create a successor with “improved character”. This task was then given to the successor agent, and so on for multiple iterations. The team measured traits such as warmth, conciseness and assertiveness across successive generations. The traits appeared to settle around fairly sensible levels after several rounds, although further investigation would be required to draw any conclusions about recursive self improvement. The experiment left me interested in how these results would change under different prompts, budgets and evaluation methods.
Offensive Cybersecurity Time Horizons
METR is a non profit research organisation that measures the capabilities and risks of frontier AI. One of its best known results is the time horizon graph, which plots the pace of AI capability improvement by benchmarking models on tasks with varying levels of difficulty, determined by how long the task would take a human expert.
On METR’s task suite, the frontier 50% success time horizon has doubled every 6-7 months, following an approximately exponential trend. The graph has become a useful reference for the rapid growth of agent capabilities, particularly on software engineering, machine learning and cybersecurity tasks.
Lyptus Research applied METR’s time-horizon method to offensive cybersecurity, finding a doubling time of 10 months across frontier models released since 2019 and 6 months for frontier models released since 2024.
The team’s latest research note shows how quickly the benchmark is being overtaken. At a two million token budget, GPT-5.5 achieved an estimated 5.1 hour time horizon and solved 80.7% of the tasks. When the team retried failures with a budget of up to 50 million tokens, overall success rose to 92.4% and the fitted horizon moved beyond the dataset’s reliable 12 hour range.
This result requires careful interpretation. The larger budget figure includes the benefit of a second attempt, and the tasks were bounded, verifiable and run against undefended targets. Even with those caveats, two points stood out to me:
Their full research note is available at: https://lyptusresearch.org/research/offensive-cyber-time-horizons
Dark AI and Cybercrime Markets
An academic researcher discussed “Dark AI” products marketed for criminal use, including names such as FraudGPT and WormGPT. These services claim to offer access to jailbroken versions of frontier models to assist in criminal activity.
Dark AI can reduce the time and skill needed to execute crimes, from crafting personalised phishing messages, creating fake identities to producing malicious code and creating toxins. Recent research into criminal AI services found that the underground LLM market is consolidating around jailbreak-as-a-service provider.
The lack of trust in criminal marketplaces creates some friction. Brands are copied, services disappear and buyers may receive a weak model or a simple wrapper instead of the capability advertised. However, this is a fragile source of protection. The best resourced groups, such as those backed by nation states, do not need to rely on such marketplaces. Established criminal operators may also be better placed to become suppliers themselves, using their reputation and resources to offer more dependable services to others.
These developments make existing fraud prevention and abuse detection controls of critical importance. AI can increase the speed, scale and sophistication of criminal activity, but attackers still need accounts, infrastructure, payment channels and access to their targets. Strong identity and access controls, anomaly detection, rate limits, API monitoring, audit logs and rapid incident response can all make that activity harder and more expensive. These controls will need to evolve as criminals find new ways to combine legitimate AI services with existing attack techniques.
Non technical sessions
Australia’s role in AI geopolitics
A recurring theme of the forum was how Australia should position itself in the international AI landscape. Several speakers argued that Australia is unlikely to be able to compete in frontier model development or advanced manufacturing, and proposed areas where Australia may have a stronger comparative advantage.
Specialising in safety evaluations. Independent evaluators are needed to test model safeguards and risks. The recently established Australian AI Safety Institute already contributes to international testing of AI agents, and building deep expertise here could make Australia a trusted source of evidence for governments, developers and organisations deciding whether a system is safe for adoption.
Building an assurance industry. Related to the above, evaluation research can contribute to guidelines for organisations deploying AI. Australia could develop expertise in auditing, certification, incident investigation and assurance, much as it has in fields such as cybersecurity, financial services and safety critical engineering. The government has already published Guidance for AI Adoption, and announced plans for Australian Standards for AI and a national values benchmark. Done well, these could form a framework for accountability, data governance, testing and human oversight in the industry adoption of AI.
Hosting compute on Australian terms. Australia is a good candidate for data centre build out given our renewable energy potential, political stability and proximity to growing markets in Asia. Simply hosting compute, however, does not guarantee broad public value. Data centres consume land, electricity and water while often employing relatively few people once construction is complete. The government’s new expectations for Australian data centres encourage operators to contribute to new energy supply, manage water sustainably, create local jobs, support research and strengthen infrastructure resilience. It has since announced plans to legislate national standards for large data centres in early 2027, including requirements concerning energy supply, water efficiency and community benefit.
Closing thoughts
The forum was a thoughtful exploration of the broader implications of AI adoption. A few concepts seem especially important in the enterprise AI context:
The highlight of the forum was the incredible energy in the conference hall. People from very different backgrounds each brought their own expertise to a problem that evolves on a timescale of months and has no guarantee of tractability. There was debate about timelines, priorities and the scale of particular risks, but also a shared willingness to examine the evidence and do the best work possible despite significant uncertainty.
Responsible AI use involves building systems that extend AI capabilities in a specific deployment while managing risks, a major part of the work of AI users. It shapes what we automate, how we measure success, how systems recover from failure and where we choose to remain accountable. We all have a role to play to ensure AI adoption goes well.
References
Program: https://www.aisafetyforum.au/program
Recorded sessions: https://www.aisafetyforum.au/videos
2026 International AI Safety Report: https://internationalaisafetyreport.org/publications