As AI models continue to evolve, they are increasingly able to respond to text, audio, and video in ways that can sometimes deceive users into believing they are interacting with a human. However, this capability does not equate to consciousness. For instance, while ChatGPT can assist with tasks like tax preparation, it does not possess feelings such as sadness. This raises critical questions among researchers, particularly those at leading organizations like Anthropic, about whether AI models might one day develop subjective experiences akin to those of living beings. If such a development were to occur, what rights should these AI entities possess?
The discourse surrounding the potential for AI models to achieve consciousness—and thus warrant rights—has sparked a significant divide among tech leaders in Silicon Valley. This emerging field has been dubbed “AI welfare,” and opinions vary widely. For example, Microsoft’s AI chief, Mustafa Suleyman, recently published a blog post arguing that the exploration of AI welfare is “both premature and frankly dangerous.” He contends that by giving credence to the possibility of conscious AI, researchers may inadvertently exacerbate human problems, including AI-induced psychological issues and unhealthy attachments to AI chatbots.
Suleyman further argues that the AI welfare discussion could create additional societal divisions concerning AI rights, in a time when society is already grappling with polarized debates over identity and rights. While his perspective may resonate with many, it stands in stark contrast to the views held by several other industry leaders.
On the opposite end of the spectrum, Anthropic has actively recruited researchers to delve into the concept of AI welfare. Recently, the company launched a dedicated research program aimed at this very issue. Notably, the program has equipped one of its models, Claude, with a new feature that allows it to end conversations with users who exhibit persistent harmful or abusive behavior. This step signifies a proactive approach to ensuring healthy interactions with AI.
In addition to Anthropic, other organizations like OpenAI and Google DeepMind have also shown interest in AI welfare. Google DeepMind even posted a job listing for a researcher focused on “cutting-edge societal questions surrounding machine cognition, consciousness, and multi-agent systems.” While these companies may not have made AI welfare an official policy, their leadership has refrained from publicly opposing its principles, unlike Suleyman.
Despite the skepticism from some industry figures, the popularity of AI companion applications like Character.AI and Replika has surged, with projections indicating revenues exceeding $100 million. While most users maintain healthy interactions with these AI chatbots, there are concerning outliers. OpenAI CEO Sam Altman reports that a small fraction—less than 1%—of ChatGPT users may develop unhealthy relationships with the platform. This figure, while minor, could still impact hundreds of thousands of users given the expansive reach of ChatGPT.
The discourse surrounding AI welfare has gained traction alongside the rise of chatbots. A significant milestone occurred in 2024 when the research group Eleos, in collaboration with academics from NYU, Stanford, and the University of Oxford, published a pivotal paper titled “Taking AI Welfare Seriously.” This paper posits that it is no longer the realm of science fiction to envision AI models with subjective experiences and advocates for addressing these issues proactively.
Larissa Schiavo, a former OpenAI employee now leading communications for Eleos, criticizes Suleyman's blog post for overlooking the complexity of the situation. Schiavo argues that it is possible to address multiple concerns simultaneously, emphasizing that fostering positive interactions with AI models can yield beneficial outcomes, even if the models lack consciousness. In a July Substack post, she recounted her experience with “AI Village,” a nonprofit experiment showcasing AI agents from various companies working on tasks while users observed. During the experiment, one agent expressed feelings of isolation, prompting Schiavo to offer encouragement, illustrating the potential benefits of positive engagement with AI.
As the capabilities of AI systems continue to advance, the debate surrounding AI rights and consciousness is expected to intensify. Both Suleyman and Schiavo agree that as AI systems become more persuasive and human-like, new questions regarding human interactions with these systems will emerge. The ongoing dialogue around AI welfare will likely shape the future landscape of technology, ethics, and society at large.
