An interview conducted by Dr. Adrian Leonard Mociulschi
Following STIRIPESURSE.RO's interview with representatives of the International Committee of the Red Cross (ICRC) on autonomous weapons and meaningful human control, we continue to examine one of the defining security challenges of the 21st century.
As advances in military artificial intelligence accelerate and international discussions on future rules and regulations gather pace, the United Nations Office for Disarmament Affairs (UNODA) offers STIRIPESURSE.RO an exclusive perspective on the security risks, legal implications and the importance of maintaining human control over the use of force.
Adrian Leonard Mociulschi: The United Nations and the International Committee of the Red Cross have repeatedly warned about the risks posed by autonomous weapons systems capable of selecting and engaging targets without meaningful human involvement.
From the perspective of the United Nations Office for Disarmament Affairs (UNODA), what are the most urgent humanitarian, legal, and security risks associated with these systems, and why is meaningful human control over the use of force considered essential for protecting human dignity, accountability, and compliance with international humanitarian law?
UNODA: At the core of the concerns surrounding autonomous weapon systems (AWS) is the human element. As autonomy in weapons increases, there is a risk that human judgement and control are progressively removed from the use of force. This is particularly concerning when decisions have life-and-death consequences.
With increasing autonomy, it can become more difficult to understand, anticipate and constrain a weapon system’s behavior and effects. This has direct implications for compliance with international humanitarian law and the protection of civilians. In complex and changing environments, systems may not reliably distinguish between combatants and civilians or account for the context and circumstances that are essential to the lawful use of force.
There is also a fundamental ethical dimension. A machine has no conscience or moral agency. It cannot make the human judgement about whether an action that is technically possible, or even legally permissible, is the right course of action. Delegating decisions over life and death to sensors, software, and machine processes therefore raises fundamental concerns for human dignity and risks crossing a line that will be difficult to reverse.
Increasing speed and autonomy can also compress decision-making timeframes, reducing opportunities for human intervention and increasing the risk of unintended or inadvertent escalation. At the international level, competition to develop increasingly autonomous capabilities could also contribute to arms-race dynamics and lower the threshold for the use of force.
The Secretary-General has been clear about the need to address these risks. He has long stressed that machines should never have the power and discretion to take human lives and has called for the urgent negotiation of a legally binding instrument containing prohibitions and regulations.
Adrian Leonard Mociulschi: Governments around the world are increasingly debating the need for international rules and governance frameworks to address autonomous weapons systems, while advances in artificial intelligence continue to accelerate.
What concrete progress has been achieved within the United Nations toward the development of international norms, rules, or legally binding instruments on autonomous weapons systems, and what are the principal obstacles that continue to hinder broader and faster international agreement on this issue?
UNODA: The international discussion has evolved significantly. What began more than a decade ago as a debate about emerging weapon technologies has increasingly become a concrete discussion about the rules and measures needed to govern them.
Within the Convention on Certain Conventional Weapons (CCW), States have considered lethal autonomous weapon systems through a Group of Governmental Experts (GGE) dedicated to the issue since 2017. The GGE has played an important role in developing common understandings of these systems and the risks they raise. Most recently, the Group has developed a set of elements of an instrument, without prejudging its legal nature. This is an important milestone: despite significant differences among States, the discussions have identified areas of common ground and produced an important body of text that can serve as a basis for further work.
The principal challenge to advancing discussions remains the diversity of views regarding the form and scope of the international response. Positions range from proposals for a new legally binding instrument containing prohibitions and restrictions, to approaches that favor non-binding measures or consider existing international humanitarian law to provide a sufficient framework.
However, as technological development, military practices and doctrines continue to evolve, the absence of agreed international limits may increase the risk of a growing gap between the pace of such developments and the evolution of the relevant regulatory and normative frameworks. In this context, establishing international regulations is essential to ensure that technological developments remain subject to effective human and legal constraints, rather than allowing practices to become entrenched before an international framework is agreed. The upcoming Seventh Review Conference of the CCW will provide an important opportunity for States to build on the common ground already established and translate it into further concrete progress by deciding on a possible new mandate for the GGE.
Adrian Leonard Mociulschi: Artificial intelligence is becoming increasingly accessible to both states and non-state actors, creating significant opportunities while also raising complex security, ethical, and governance challenges. As these technologies continue to evolve, many citizens around the world are following these developments with both hope and concern.
What message would UNODA like to convey to citizens around the world regarding the future governance of military applications of artificial intelligence, and what measures are most important to ensure that technological innovation remains consistent with international security, human responsibility, and the long-term pursuit of peace?
UNODA: The most important message is that innovation in the military domain needs direction. Artificial intelligence can bring important benefits to international peace and security — from peacekeeping to the verification of disarmament agreements. The challenge is to ensure that competition to develop those capabilities does not outpace our ability to govern them responsibly and manage the risks they may create.
In that sense, good governance should not be seen as a constraint on innovation. Establishing appropriate safeguards and guardrails can create the conditions for better innovation. Clear rules, limits, boundaries, and common expectations can give governments and the private sector greater understanding about what systems should be designed to do and what they should not do.
Governance can also create incentives for investments in system safety and security measures, such as reliability, human control and oversight, testing and evaluation, to name just a few.
The General Assembly has already recognized the importance of a whole-life-cycle approach to AI — ranging from pre-design to decommissioning of these systems. The life cycle approach is a good example of how governance can be considered early (at the pre-design stage) and incorporated into systems before they are designed, procured, and deployed in operational environments.
International cooperation is important. We need cooperation among States, including from the States that are developing advanced AI capabilities, but particularly representatives from developing countries. The consequences of these technologies will not remain confined to the States that developed them; their deployment will impact security elsewhere in the world. The General Assembly has taken important first steps on this front. As the world’s most universal intergovernmental body, it is important that it continues to be at the heart of global discussions on AI in the military domain.
Ultimately, UNODA’s concern is not whether AI will become more capable. It is whether increasing capability also increases risks to international peace and security. Our objective should be to uphold international law, including international humanitarian law and international human rights law, while ensuring that human responsibility, judgement, and agency remain at the centre of military applications of AI. Ultimately, humans must retain agency over AI systems — not the other way around.
Our sincere thanks to the United Nations Office for Disarmament Affairs (UNODA) for its cooperation and assistance in facilitating this discussion on military applications of artificial intelligence, human control over the use of force, and the regulation of autonomous weapon systems.
An interview conducted by Dr. Adrian Leonard Mociulschi
