In a troubling discovery for businesses racing to automate customer service, OpenAI’s popular Whisper transcription software has been caught adding fabricated text to conversations, including potentially harmful content that speakers never uttered.
The findings underscore a growing challenge for companies betting billions on AI to handle sensitive customer interactions, from call centers to medical offices. Even advanced language models continue to “hallucinate” or generate false information that could damage customer relationships or create legal liability.
“If a chatbot is providing shoppers with wrong or misleading answers and advice, it can be a big problem and might hurt the organization’s reputation,” Iris Zarecki, director of product marketing at AI company K2view, told PYMNTS. “A couple of recent examples of this include Air Canada’s chatbot giving incorrect information to a traveler and arguing the chatbot is “responsible for its own actions,” and an AI-powered chatbot created by New York City to help small business owners giving advice that misstates local policies and advised companies to violate the law.”
The Associated Press reported that OpenAI’s Whisper stumbles on a basic task: accurately writing down what people say. Researchers found the AI transcription tool frequently invents text out of thin air, including disturbing content about violence and race. One study found hallucinations in 8 out of 10 transcripts. The glitch raises alarm bells as hospitals and clinics rush to adopt Whisper-based tools for patient consultations despite OpenAI’s explicit warnings against using it for high-risk scenarios.
Hallucinations in AI are defined as confident but inaccurate responses that are not justified by the training data used for the AI model, Chris Sheehan, senior vice president and general manager of strategic accounts at Applause, a company that does generative AI testing among other services, told PYMNTS. He said that
AI hallucinations occur when generative AI models create outputs that are nonsensical or inaccurate. Applause’s 2024 Generative AI survey found that 37% of respondents had experienced hallucinations.
“Presenting inaccurate or offensive information to customers is annoying and time-consuming, but more deeply, it erodes trust in the brand that it could deploy technology that was obviously not properly tested and vetted,” Sheehan said.
While both agentic AI and chatbots can hallucinate, it’s important to distinguish between them, Dan Balaceanu, chief product officer at Druid AI, told PYMNTS. Chatbots enable natural interaction via text, voice, or both but are typically rule-based and only respond to specific keywords or phrases. On the other hand, AI agents can handle more complex conversations and provide personalized solutions to customers. These agents leverage advanced AI techniques like natural language processing and machine learning to provide customized answers to queries.
“AI hallucinations are a form of software application defect or error,” he added. “These are more problematic, though, because you can’t acknowledge them easily. You don’t get a blue screen or an error message in front of you. Instead, you get a nice and apparently correct response, which can be incredibly misleading.”
However, Cordial’s Chief Product and Engineering Officer Matt Howland sees an upside to hallucinations. He told PYMNTS that AI’s creative departures from reality can be both a challenge and an opportunity. These quirks show up when AI models work with limited data or unclear instructions, sometimes leading to content ranging from mildly inaccurate to completely fantastical.
“What fascinates me is how marketers and brands can actually turn this seeming limitation into an advantage,” he added. “While we absolutely need accuracy for technical documentation and customer communications, these unexpected AI behaviors can spark creative breakthroughs in marketing campaigns and content creation. The key lies in knowing when to rein in AI’s imagination and when to let it roam free. With this, everyone should treat AI as a creative collaborator rather than a production tool.”
For all PYMNTS AI coverage, subscribe to the daily AI Newsletter.
We’re always on the lookout for opportunities to partner with innovators and disruptors.
Learn More