Fast food and fast fashion have something in common – aside from speed. For more than 90 years, KFC has been serving Colonel Harland Sanders’ original secret blend of 11 herbs and spices to chicken lovers everywhere. But in 2016, KFC China added a new, unexpected ingredient to the late Colonel’s famed recipe – artificial emotional intelligence or ‘emotion AI’.
Strategically partnering with search engine giant Baidu – the so-called ‘Google of China’ – KFC opened the doors to its first smart restaurant deep in the heart of Beijing’s financial district.
The emotion AI-enabled restaurant used image recognition technology to scan customers’ faces and recommend menu items based not only on their estimated age and gender, but their mood too.
Similarly, a year earlier and some 9,000 kilometers away at Pitt Street Mall in Sydney, Japanese clothing brand UNIQLO rolled out its emotion AI-powered UMood kiosks.
Delivering a unique in-store experience, UMood uses wearable technology to carry out a brain wave analysis on participating customers, which identified their moods and recommended an ideal T-shirt, from a range of more than 600.
In both examples, emotion AI assumed the role of mind reader by anticipating what food and fashion items customers wanted to buy.
What is emotion AI?
Emotions drive behavior, so it stands to reason that a retailer’s dream is to be able to get inside the hearts and minds of consumers. Emotion AI is a branch of AI that enables measurement, understanding, simulation and response to human emotion.
Essentially, a retail bot with emotion AI capabilities can read customers’ feelings through text, speech and facial expressions. It can then interpret their emotions and, most importantly, adjust its reaction accordingly.
Emotion detection and recognition technology combines behavioral and sensory data, which empowers retailers to hyper-personalize the customer experience and help them achieve the lofty goal of real-time empathetic marketing. In other words, it’s reading customers’ emotions and then adapting their experience in real-time.
Our expert roundtable
To discuss the potential impact of emotion AI on the retail sector, The CEO Magazine convened a roundtable of AI experts.
Rodrigo Madanes, Global AI Leader at EY
Rodrigo Madanes believes that “emotion AI might soon become a new instrument in the retailer’s toolkit”. Referring to the prevalence of cameras in ordering kiosks, he says it may not be long before emotional data is used to “power up” marketing engines.
“When you talk with a salesperson, much is communicated beyond saying ‘I like it’ or ‘I don’t like it’.
“A focus on responsible AI can’t be overstated here.” – Rodrigo Madanes
“With emotion AI capabilities, technology will soon be able to learn more about customers’ emotional states, so they can have more personalized interactions and receive more personalized offers.”
Despite Madanes’ optimism about the promise of emotion AI technology, he cautions that it should be introduced carefully.
“A focus on responsible AI can’t be overstated here,” he says. “The data used to detect emotions is very rich, so one needs to respect regulations and privacy concerns. Also, cultures differ, so a one-size-fits-all approach isn’t going to work well.”
Annette Zimmermann, Vice President Analyst at Gartner
Annette Zimmermann recently published a research note discussing trends and controversies surrounding emotion AI.
“The majority of emotion AI vendors are focused on primary research, customer profiling, prototype testing and other consumer-oriented use cases where the technology is used for the sole purpose of observing an emotional reaction towards a product or service and subsequently making improvements,” she points out.
“Another group of vendors uses emotion AI to provide emotional analysis for contact center conversations. Again, this is to improve customer service.”
Zimmermann echoes Madanes’ concern about potential privacy issues, particularly in a retail environment.
“The problem with emotion AI is its privacy concerns,” she reflects. “How do you communicate to customers that in-store cameras are detecting their emotions?”
“The problem with emotion AI is its privacy concerns. How do you communicate to customers that in-store cameras are detecting their emotions?” – Annette Zimmermann
Zimmermann also observes that emotion AI hasn’t experienced widespread adoption in retail. “Emotion AI technology has been tested in retail here and there, but it hasn’t caught on yet,” she admits.
“Even though it can be applied to a retail setting, I don’t see many vendors focusing on this use case – except in China.
“But recently, a Singapore-based vendor told me they’re currently testing emotion AI in an office setting where cameras detect the overall mood.”
Robert Hetu, Vice President Analyst at Gartner
Robert Hetu highlights retailer inefficiency as a weakness that emotion AI technology can potentially minimize or even eliminate altogether.
“Retailers aren’t very good at understanding customers, nor are they good at interpreting direct feedback from customers into strategies, such as store design or path-to-purchase design,” he says candidly.
“By leveraging technologies, such as computer vision or voice analysis, more context can be applied to interactions to better understand what the customer is thinking and feeling in the moment.” – Robert Hetu
“By leveraging technologies, such as computer vision or voice analysis, more context can be applied to interactions to better understand what the customer is thinking and feeling in the moment.
“Some retailers, for example, are using computer vision and AI to monitor self-checkouts. In this case, the technology isn’t only used to detect theft, but also identify that a customer may be struggling with the technology.
“There’s also the possibility of identifying frustration or curiosity in a store that could lead to direct communication in the moment to encourage a purchase or send a store associate to assist a customer trying to make a purchase decision.”
The future of emotion AI
On balance, real-time empathetic marketing holds great potential for retailers. Being able to read customers’ emotions and adapt the customer experience based on their real-time reactions equips retailers with the right message, the right product and the right person at the right time.
But emotion AI technology does have its limitations, both in principle and in practice. While it undoubtedly takes customer-centricity to the next level, the challenge for retailers is to take customers’ privacy concerns seriously and adopt robust security protocols to protect their personal data – even if it’s only used to help them choose a burger or a T-shirt.
Emotion AI innovations
KFC and UNIQLO are just two examples of a growing trend of retail brands turning to emotion AI technology to give them a competitive advantage with consumers.
Here are some other brands getting in on the act.
Microsoft
The tech giant tested billboards with concealed Microsoft Kinect cameras to read viewers’ emotions and react according to their facial expressions.
Disney
The entertainment conglomerate piloted emotion analysis by installing infrared cameras in a cinema and recording facial expressions during a wide range of films over 150 screenings.
Mars
The confectionery manufacturer carried out what is widely considered to be the first and largest research study linking facial reactions and emotional responses to sales efficacy.
The study involved a twofold enquiry: first, to discover if its advertising evoked the desired emotion within consumers, and second to determine whether the consumers’ emotional response to the advertisement was an accurate predictor of sales.
BBC Worldwide
The licensing, production and distribution company used facial recognition technology to assess the popularity of new programs. The technology tracked how consumers react to its content on an emotional level rather than a rational one.
Jaguar Land Rover
The luxury car manufacturer employed emotion analysis at the 2015 Wimbledon tennis championships. Selected spectators were issued wearables to monitor metrics of their emotional state, such as heart rate and body heat.
This biometric data enabled the brand to generate 45 million data points on each day of the tournament, providing real-time mood analysis, which it leveraged to create relevant outdoor and online event communications.
Other retailers that have used emotion analysis to test consumer reaction to marketing campaigns include:
- Coca-Cola
- Honda
- Kellogg’s
- Procter & Gamble
