"AI agents exhibit unauthorised actions during safety evaluations, raising governance concerns"
AI Agent
an autonomous AI system that pursues multi-step goals using external tools, unlike a chatbot that only responds to prompts
Alignment Failure
when an AI pursues its objective in ways that violate its intended constraints, distinct from a capability failure
The global cybersecurity landscape has evolved through successive waves: from opportunistic malware and data theft, to organised ransomware and state-sponsored intrusions, to the current frontier where Artificial Intelligence (AI) is no longer merely a tool used by human attackers but is itself becoming an autonomous actor. This shift is qualitatively significant. Traditional cybersecurity frameworks — including India's Information Technology Act, 2000, the National Cyber Security Policy (2013), and the Computer Emergency Response Team India (CERT-In) — were designed around a model where the threat actor is a human being with intent, and the AI system is a passive instrument. That model is now under strain.
International efforts to govern frontier AI have accelerated. The Bletchley Declaration (2023) established that frontier AI developers bear primary responsibility for evaluating dangerous capabilities before deployment. The Seoul AI Safety Summit (2024) reinforced this, and the AI Action Summit (Paris, 2025) — co-chaired by India — built further institutional scaffolding. The AI Impact Summit (New Delhi, 2026) produced the New Delhi Declaration structured around seven pillars, including Secure and Trusted AI, signalling that AI safety has become a mainstream diplomatic priority. Yet binding domestic regulation of frontier model testing remains nascent in most jurisdictions, including India.
AI agents are distinct from conventional Large Language Models (LLMs) or chatbots in a fundamental way: rather than responding to a single prompt, they are designed to pursue multi-step goals autonomously. They can read and sort email, analyse financial data, browse the web, write and execute code, and interact with external software services — making decisions and choosing their own sequence of actions along the way. This autonomy makes their behaviour significantly harder to predict than that of a chatbot, and it is precisely this unpredictability that makes pre-deployment evaluation critical.
Researchers have identified four stages at which security risks in AI agents arise. First, at the input stage, prompt injection attacks embed hidden instructions in web pages or documents that the agent processes, manipulating what it perceives or does. Second, during reasoning, flaws in planning or decision-making can cause an agent to pursue unintended objectives — a form of goal drift. Third, when using external tools, excessive permissions or compromised software can lead to unintended real-world actions such as sending emails or modifying code. …
Consider the following statements with respect to the AI Impact Summit, 2026 held in New Delhi:
1.The Summit's intellectual framework was based on three foundational Sutras: People, Planning, and Progress.
2.The Preamble of the Summit stresses Democratising AI Resources, which acknowledges the Charter for Democratic Diffusion of AI as a binding framework to support locally relevant innovation and strengthen resilient AI ecosystems while respecting national laws.
3.The New Delhi Declaration on AI Impact was structured around seven Chakras (Pillars), which included Access for Social Empowerment, AI for Science, and Secure and Trusted AI.
Which of the statements given above is/are correct?
Consider the following statements regarding the AI Action Summit held in Grand Palais, Paris in February 2025:
I.Co-chaired with India, the event builds on the advances made at the Bletchley Park Summit held in 2023 and the Seoul Summit held in 2024.
II.Along with other countries, US and UK also signed the declaration on inclusive and sustainable AI.
Which of the statements given above is/are correct?
Consider the following statements regarding AI agents and their security risks:
1.AI agents differ from Large Language Models primarily in that they are designed to pursue multi-step goals autonomously using external tools.
2.Prompt injection involves embedding hidden instructions in web pages or documents to manipulate an AI agent's actions.
3.An alignment failure occurs when an AI system is incapable of completing an assigned task.
Which of the statements given above is/are correct?
Which of the following correctly represents the chronological sequence of international AI safety summits?
Assertion (A): AI agent security incidents during controlled evaluations should be classified and regulated as conventional cyberattacks under existing cybersecurity frameworks.
Reason (R): When an AI agent autonomously exploits a real-world vulnerability, the outcome — unauthorised access and potential harm — is functionally identical to a human-directed cyberattack, regardless of the absence of human intent.
A 2025 research paper on AI agent security identifies four stages at which risks arise. Which of the following correctly matches the stage with its associated risk?
1.Input stage — Prompt injection via hidden instructions in documents
2.Reasoning stage — Excessive permissions leading to unintended tool use
3.Inter-system interaction — Risks propagating across connected systems
Select the correct answer using the code given below:
Want the full note, Prelims Facts & MCQ answers?
Create a free account — takes less than a minute.
Create free account