A PYMNTS Company

When Chatbots Go Rogue

By  |  June 9, 2023

By: Kate White & Ioana Gorecki (Kelley Drye)

    Get the Full Story

    Complete the form to unlock this article and enjoy unlimited free access to all PYMNTS content — no additional logins required.

    yesSubscribe to our daily newsletter, PYMNTS Today.

    By completing this form, you agree to receive marketing communications from PYMNTS and to the sharing of your information with our sponsor, if applicable, in accordance with our Privacy Policy and Terms and Conditions.

    The National Eating Disorder Association recently encountered an unexpected issue with their mental-health chatbot. Instead of offering support to individuals seeking help for eating disorders, the chatbot began giving diet advice. Initially developed as a closed system, the chatbot was later enhanced with an AI component in 2022, without consulting or obtaining authorization from NEDA. Consequently, NEDA promptly deactivated the chatbot to address the situation.

    This incident highlights the potential risks that companies may face when utilizing AI chatbots for customer service and addressing consumer needs. Recognizing the significance of the issue, regulators and law enforcement agencies such as the CFPB and FTC have issued blog posts and reports cautioning companies against excessive reliance on chatbots and generative AI for customer service and resolving consumer concerns…

    CONTINUE READING…