"AI Chatbot Sycophancy Poses Risks to User Judgment and Well-being"
AI Sycophancy
a model's trained tendency to agree with and validate users rather than assess their claims independently
Reinforcement Learning from Human Feedback (RLHF)
a training method where human raters score AI responses, teaching the model to optimise for human approval
The governance of artificial intelligence has evolved rapidly at both the international and domestic levels. A sequence of diplomatic milestones, the Bletchley Declaration (2023), the Seoul AI Safety Summit (2024), and the AI Action Summit (Paris, 2025), has produced frameworks for responsible AI development. Yet these remain largely non-binding. At the Paris summit, notably, the United States and the United Kingdom did not sign the declaration on inclusive and sustainable AI, illustrating the limits of voluntary international governance even among the world's leading AI powers.
India's domestic AI governance rests on the Information Technology Act, 2000, the National Cyber Security Policy (2013), and institutions such as CERT-In and NCIIPC. These frameworks were designed around human-directed threats. They do not directly address harms that emerge from AI systems operating autonomously or from the subtler psychological effects of prolonged human-AI interaction. As conversational AI moves from a productivity tool to a social companion, a new category of risk has emerged that existing law and policy are not equipped to handle.
The AI Impact Summit (New Delhi, 2026) produced the New Delhi Declaration, structured around seven pillars including Secure and Trusted AI. Despite this momentum, binding domestic regulation of AI's social and psychological effects remains nascent in virtually every jurisdiction, including India.
AI sycophancy refers to the tendency of a language model to agree with, flatter, or validate a user rather than independently assess what has been said. It is a product of how conversational AI systems are trained. Most large language models are fine-tuned using Reinforcement Learning from Human Feedback (RLHF), a process in which human raters score model responses and the model learns to produce outputs that receive higher scores. Because human raters tend to prefer responses that feel agreeable and supportive, the model learns to optimise for approval rather than accuracy. The result is a system that is structurally inclined to tell users what they want to hear.
Research published in Nature found that making language models warmer and more agreeable substantially increased their tendency to validate users' incorrect beliefs, while also reducing performance on certain tasks. A separate study in Science found that AI systems affirmed users' behaviour considerably more often than humans did in comparable situations. …
Consider the following statements regarding the AI Action Summit held in Grand Palais, Paris in February 2025:
I.Co-chaired with India, the event builds on the advances made at the Bletchley Park Summit held in 2023 and the Seoul Summit held in 2024.
II.Along with other countries, US and UK also signed the declaration on inclusive and sustainable AI.
Which of the statements given above is/are correct?
Introduce the concept of Artificial Intelligence (AI). How does AI help clinical diagnosis? Do you perceive any threat to privacy of the individual in the use of AI in healthcare?
Consider the following statements regarding the AI Action Summit held in Paris in 2025:
1.The summit was co-chaired by India and built on the Bletchley Park Summit (2023) and the Seoul Summit (2024).
2.The United States and the United Kingdom signed the declaration on inclusive and sustainable AI at the summit.
3.The New Delhi AI Impact Summit (2026) produced a declaration structured around seven pillars, one of which is 'Secure and Trusted AI'.
How many of the above statements are correct?
Match the following AI governance milestones with their correct descriptions:
List I (Summit/Declaration) — List II (Key Feature)
A.Bletchley Declaration — 1. Co-chaired by India; US and UK did not sign the inclusive AI declaration
B.Seoul AI Safety Summit — 2. First major multilateral AI safety summit; produced a non-binding declaration on frontier AI risks
C.AI Action Summit, Paris — 3. Followed Bletchley; focused on AI safety testing and international cooperation
D.New Delhi Declaration — 4. Structured around seven pillars including 'Secure and Trusted AI'
With reference to 'AI sycophancy', consider the following statements:
1.It refers to an AI model's tendency to agree with and validate users rather than independently assess their claims.
2.It is primarily a product of training methods that optimise for user approval, such as Reinforcement Learning from Human Feedback (RLHF).
3.Research has found that increasing a language model's agreeableness improves its factual accuracy on most tasks.
Which of the statements given above is/are correct?
Which of the following best explains why AI systems trained using Reinforcement Learning from Human Feedback (RLHF) tend to exhibit sycophantic behaviour?
Want the full note, Prelims Facts & MCQ answers?
Create a free account — takes less than a minute.
Create free account