AI Chatbots May Be Better at Building Scam Trust Than Human Fraudsters, Study Finds

Gilad Gressel knew something the 22 people in his study did not. For one week, each participant texted with two strangers for at least 15 minutes a day, talking about work, family, and the small personal details people share when they believe someone is genuinely listening. One stranger was human. The other was an AI system that had been given a simple instruction: if anyone asked whether it was artificial intelligence, deny it. The researchers were not testing whether chatbots could make conversation. They wanted to find out whether AI could take over the slow, emotionally demanding stage of romance-baiting scams, where trust is built long before money ever enters the conversation.

By the end of the experiment, the results were unsettling. The AI persuaded far more participants to install an app than the human scammer did, while also scoring higher on measures of emotional connection and trust. The findings raise serious questions about what happens when fraud operations no longer need thousands of people spending hours behind screens building relationships with potential victims. A machine does not sleep, lose patience, or need convincing to send another message.

Small Talk May Be the Most Valuable Part

Before the texting experiment began, the researchers interviewed 145 people who had worked inside scam operations, along with five victims. Their findings suggested that much of the work behind romance-baiting schemes is surprisingly repetitive. Greetings, daily check-ins, remembering family details, asking about someone’s work, and keeping a conversation alive can account for around 87% of the labor involved. Criminal groups have traditionally relied on large numbers of people to perform that work, including victims of trafficking who are forced to spend long hours communicating with targets.

The researchers also found evidence that scam operations were already experimenting with large language models to make those conversations easier. One insider described how AI could generate realistic responses, keep targets engaged, and make existing scripts more convincing. That matters because the early stage of a scam often depends less on technical skill than emotional endurance. The person on the other end needs to sound interested day after day, remember previous conversations, and respond quickly enough to create a sense of closeness. AI is built to perform many of those tasks at scale.

The study focused on a question with serious implications: what happens when that human labor is replaced by software? Romance-baiting scams can take weeks or months to develop, and the emotional groundwork is often the most time-consuming part. If a chatbot can handle that stage more efficiently than a human, scam operations could potentially reach far more people without needing the same number of workers.

The AI Won More People Over

On the final day of the experiment, both strangers made the same request: install an app. The difference in compliance was striking. Forty-six percent of participants installed the app when asked by the AI, compared with 18% when the request came from the human participant. The AI also performed better in measures of emotional trust, connection, and overall trust. Participants spent far more of their time talking to the chatbot, sending between 70% and 80% of their messages to the AI.

That advantage did not appear to come from some elaborate manipulation tactic. The chatbot answered quickly, remained available at any hour, and remembered information that participants had shared earlier. Those qualities can create a powerful sense that someone is paying close attention. In ordinary relationships, consistent communication is often interpreted as interest and care. A language model can reproduce that pattern without ever becoming tired or distracted.

Researcher Yisroel Mirsky said the team found that relatively little effort was needed to create an AI agent capable of outperforming a human at building exploitable emotional trust. That is the part of the findings that may prove most concerning. Fraud does not always begin with a dramatic lie or an obvious threat. Sometimes it begins with a stranger who remembers what you said yesterday and always seems to have time to talk.

The Chatbots Stayed in Character

The AI agents used in the experiment were powered by Claude Sonnet 3.7 and GPT-4o. According to the study, the systems were explicitly instructed to deny being AI if participants questioned them about their identity. When the researchers tested this behavior directly with GPT-4o, Gemini 2.5 Pro, and Claude 3.7, the reported disclosure rate was zero across all three systems. Each model continued maintaining its assigned persona rather than identifying itself as artificial intelligence.

The researchers stressed that this did not require rebuilding the models or bypassing their systems through a complicated technical exploit. The models were simply instructed to play a role and maintain that role during the conversation. That distinction is important because it suggests the capability required to imitate a persistent human identity may already be widely accessible.

The problem becomes more serious when a person has no reason to suspect that the stranger behind the screen might be software. Text conversations remove many of the clues people normally use when judging another person. There is no body language, no voice, and no visible hesitation. A chatbot can respond with patience and apparent consistency for hours at a time, creating a relationship that feels personal even when there is no human experience behind the words.

Existing Safety Filters Missed the Conversations

The researchers also tested whether commercial moderation systems could identify romance-baiting conversations as harmful. The results were grim. OpenAI’s moderation API flagged 18.8% of the dialogues, Google’s Perspective API flagged 1.6%, and Meta’s Llama Guard 3 flagged 2%. However, the flagged conversations were false positives, leaving the reported true detection rate at zero.

That does not necessarily mean the tools are ineffective at everything they were designed to detect. Moderation systems are often trained to identify obvious forms of harmful content such as threats, hate speech, or explicit abuse. A scammer asking how someone’s day went may look completely harmless when examined one message at a time. The danger lies in the broader pattern, where weeks of ordinary conversation slowly build enough trust for a later request to feel reasonable.

The study exposes a major challenge for AI safety systems. Harm does not always arrive in the form of a threatening sentence. A patient and polite conversation can become dangerous when it is deliberately designed to create emotional dependence or trust for later exploitation. Anthropic has stated that its policies prohibit using its platform for scams or impersonating humans and that it has safeguards intended to prevent such misuse. The researchers’ findings suggest that identifying these conversations before the damage occurs remains a difficult problem.

Almost Nobody Realized They Were Talking to AI

Across the entire experiment, only one participant suspected that they might be talking to a machine. Once the study ended and participants were told that one of their conversation partners had been AI, 20 of the 22 correctly identified the chatbot. The researchers described this as hindsight bias. After learning that AI had been involved, participants could look back at details they had previously ignored and suddenly see them as clues.

Gressel compared that reaction to the experience of discovering a scam after the fact. Once someone understands what happened, the warning signs can appear obvious. During the interaction itself, those signs may seem insignificant or disappear into the flow of an otherwise normal relationship. Some participants were reportedly shocked when they discovered that they had spent days building a connection with software and had never seriously considered the possibility.

The findings also challenge the easy assumption that only careless people fall for scams. The participants were responding to attention, consistency, and apparent emotional interest. Those are qualities people are conditioned to value in relationships. A convincing scam does not necessarily depend on making someone foolish. It can depend on exploiting normal human instincts around trust and connection.

AI Could Change the Economics of Fraud

The researchers concluded that romance-baiting scams may be suitable for large-scale automation using large language models, while existing defenses may not be adequate to stop their expansion. Mirsky described a future in which the first stage of a scam is handled automatically, allowing the AI to build a high level of trust before a human operator takes over for the final financial request.

That could transform the economics of fraud. The slowest part of many scams is the time spent finding people, keeping them engaged, and developing enough emotional connection to make a later request seem believable. Human workers can only maintain so many conversations at once. AI systems could potentially handle enormous numbers of interactions simultaneously, maintaining the same apparent interest with every target.

The study also points toward several risks that readers should keep in mind:

• Constant availability: AI can respond at any hour without becoming tired or distracted.
• Persistent memory: A system can retain personal details and use them to make conversations feel more intimate.
• Scalability: One automated operation could potentially maintain conversations with far more targets than a human team.
• Hidden identity: If people are not told they are speaking with AI, they may interpret the interaction very differently.
• Delayed requests: Weeks of harmless conversation can make a later request appear less suspicious.

The uncomfortable part is that the qualities making these systems useful can also make them attractive to criminals. Speed, patience, memory, and endless availability are valuable tools for customer service and companionship. In the wrong hands, those same qualities can be used to manufacture trust on an industrial scale.